1. Py-Feat basics

1. Py-Feat basics

Written by Eshin Jolly

This tutorial goes over the basics of using Py-Feat’s API. You can try it out interactively in Google Collab: Open In Colab

At a broad level you will be working with 2 main objects in Py-Feat:

# Uncomment the line below and run this only if you're using Google Collab
# !pip install -q py-feat

Detectors

A detector is a swiss-army-knife class that “glues” together a particular combination of a Face, Landmark, Action Unit, and Emotion detection model into a single object. This allows us to provide a very easy-to-use high-level API, e.g. detector.detect_image('my_image.jpg'), which will automatically make use of the correct underlying model to solve the sub-tasks of identifying face locations, getting landmarks, extracting action units, etc.

from feat.detector import Detector

# The verbose flag will print out messages indicating whether a pre-trained model
# is being downloaded for the first time, or being loaded from disk.
# In this case I already have all the models downloaded, but if you run this notebook on
# your personal computer or Google Collab, these will be downloaded the first time you
# use a Detector.
detector = Detector(verbose=True)

detector
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/mobilenet0.25_Final.pth
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/mobilenet_224_model_best_gdconv_external.pth.tar
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/DRMLNetParams.pth
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/ResMaskNet_Z_resmasking_dropout1_rot30.pth
feat.detector.Detector(face_model=retinaface, landmark_model=mobilenet, au_model=drml, emotion_model=resmasknet, facepose_model=pnp)

After initializing a detector you can easily swap one or more underlying models using the .change_model method. You can also disable any models by setting them to None:

detector.change_model(
    face_model="MTCNN", emotion_model="svm", au_model="svm", landmark_model=None
)
detector
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/onet.pt
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/pnet.pt
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/rnet.pt
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/hog_pca_all_emotio.joblib
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/hog_pca_all_emotio.joblib
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/hog_scalar_aus.joblib
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/svm_568.joblib
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/hog_pca_all_emotio.joblib
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/hog_scalar_aus.joblib
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/emoSVM38.joblib
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/emo_hog_pca.joblib
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/emo_hog_scalar.joblib
feat.detector.Detector(face_model=mtcnn, landmark_model=None, au_model=svm, emotion_model=svm, facepose_model=pnp)

You’ll primarily use a detector instance by calling its .detect_image() or .detect_video() methods. Both methods take as input a filename (or list of filenames) and return a Fex data class instance. Detectors also have lower-level detection methods like .detect_face() and .detect_landmarks() which operate on and return numpy arrays directly.

Fex data classes

A Fex data class is just a special type of pandas dataframe then makes it easier to work with the results returned by a detector. Each row contains information about a detected face and each column contains the output of that detection (e.g. x, y location; emotion) or some meta-data about the input file (e.g. the filename).

Fex data instances have helper methods on them to quickly retrieve the appropriate data you want without having to search through column names yourself, e.g. fex.emotions(). They also have methods for plotting e.g. fex.plot_detections(), signal-processing e.g. fex.downsample(), and statistical analysis e.g. fex.regress().

from feat.data import Fex
import pandas as pd

fex = Fex()

isinstance(fex, pd.DataFrame)
True

More often than not, you’ll be using Fex data classes that are return by Detector rather than creating them from scratch. Check out the next tutorial for more details on how this works!