1. Py-Feat basics#

Written by Eshin Jolly

This tutorial goes over the basics of using Py-Feat’s API. You can try it out interactively in Google Collab: Open In Colab

At a broad level you will be working with 2 main objects in Py-Feat:

# Uncomment the line below and run this only if you're using Google Collab
# !pip install -q py-feat

Detectors#

A detector is a swiss-army-knife class that “glues” together a particular combination of a Face, Landmark, Action Unit, and Emotion detection model into a single object. This allows us to provide a very easy-to-use high-level API, e.g. detector.detect_image('my_image.jpg'), which will automatically make use of the correct underlying model to solve the sub-tasks of identifying face locations, getting landmarks, extracting action units, etc.

The first time you initialize a Detector instance on your computer will take a moment as Py-Feat will automatically download required pretrained model weights for you and save them to disk. Everytime after that it will use existing model weights:

from feat.detector import Detector

# The verbose flag will print out messages indicating whether a pre-trained model
# is being downloaded for the first time, or being loaded from disk.
# In this case I already have all the models downloaded, but if you run this notebook on
# your personal computer or Google Collab, these will be downloaded the first time you
# use a Detector.
detector = Detector(verbose=True)

detector

Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/mobilenet0.25_Final.pth
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/mobilenet_224_model_best_gdconv_external.pth.tar
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/svm_60_Nov22022.pkl
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/upper_face_pcaSet.pkl
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/lower_face_pcaSet.pkl
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/full_face_pcaSet.pkl
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/ResMaskNet_Z_resmasking_dropout1_rot30.pth
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/ResMaskNet_fer2013_config.json
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/reference_3d_68_points_trans.npy
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/img2pose_v1.pth
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/WIDER_train_pose_mean_v1.npy
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/WIDER_train_pose_stddev_v1.npy

feat.detector.Detector(face_model=retinaface, landmark_model=mobilenet, au_model=svm, emotion_model=resmasknet, facepose_model=img2pose)

Default detectors#

The default models loaded by Py-Feat are the currently recommended ones for each detection task. You see a full list of supported models here and instructions for contributing a new detector here. The current defaults include:

  • Face Model: retinaface

  • Landmark Model: mobilefacenet

  • AU Model: svm

  • Emotion Model: resmasknet

  • Face Pose Model:img2pose

After initializing a detector you can easily swap one or more underlying models using the .change_model method. All models names should be provided as strings and are case-insensitive:

detector.change_model(face_model="MTCNN")
detector

Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/onet.pt
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/pnet.pt
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/rnet.pt
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/mobilenet_224_model_best_gdconv_external.pth.tar
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/svm_60_Nov22022.pkl
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/upper_face_pcaSet.pkl
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/lower_face_pcaSet.pkl
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/full_face_pcaSet.pkl
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/ResMaskNet_Z_resmasking_dropout1_rot30.pth
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/ResMaskNet_fer2013_config.json
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/reference_3d_68_points_trans.npy
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/img2pose_v1.pth
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/WIDER_train_pose_mean_v1.npy
Using downloaded and verified file: /Users/Esh/Documents/pypackages/py-feat/feat/resources/WIDER_train_pose_stddev_v1.npy
Changing face_model from retinaface -> mtcnn

feat.detector.Detector(face_model=mtcnn, landmark_model=mobilenet, au_model=svm, emotion_model=resmasknet, facepose_model=img2pose)

You’ll primarily use a detector instance by calling its .detect_image() or .detect_video() methods. Both methods take as input a filename (or list of filenames) and return a Fex data class instance. Detectors also have lower-level detection methods like .detect_face() and .detect_landmarks() which operate on and return numpy arrays directly.

Fex data classes#

A Fex data class is just a special type of pandas dataframe then makes it easier to work with the results returned by a detector. Each row contains information about a detected face and each column contains the output of that detection (e.g. x, y location; emotion) or some meta-data about the input file (e.g. the filename).

Fex data instances have helper methods on them to quickly retrieve the appropriate data you want without having to search through column names yourself, e.g. fex.emotions(). They also have methods for plotting e.g. fex.plot_detections(), signal-processing e.g. fex.downsample(), and statistical analysis e.g. fex.regress().

from feat.data import Fex
import pandas as pd

fex = Fex()

isinstance(fex, pd.DataFrame)

True

More often than not, you’ll be using Fex data classes that are return by Detector rather than creating them from scratch. Check out the next tutorial for more details on how this works!