Contents

API Reference

API Reference

This reference provides detailed documentation for all the features in FEAT

feat.detector module

Main Detector class. The Detector class wraps other pre-trained models (e.g. face detector, au detector) and provides a high-level API to make it easier to perform detection

class feat.detector.Detector(face_model='retinaface', landmark_model='mobilenet', au_model='rf', emotion_model='resmasknet', facepose_model='pnp', n_jobs=1, verbose=False)

Bases: object

change_model(**kwargs)

Swap one or more pre-trained detector models for another one. Just pass in the the new models to use as kwargs, e.g. emotion_model=’rf’

detect_aus(frame, landmarks)

Detect Action Units from image or video frame

Parameters

  • frame (np.ndarray) – image loaded in array format (n, m, 3)

  • landmarks (array) – 68 landmarks used to localize face.

Returns

Action Unit predictions

Return type

array

Examples

>>> from feat import Detector
>>> from feat.utils import read_pictures
>>> frame = read_pictures(['my_image.jpg'])
>>> detector = Detector()
>>> detector.detect_aus(frame)
detect_emotions(frame, facebox, landmarks)

Detect emotions from image or video frame

Parameters

  • frame ([type]) – [description]

  • facebox ([type]) – [description]

  • landmarks ([type]) – [description]

Returns

Action Unit predictions

Return type

array

Examples

>>> from feat import Detector
>>> from feat.utils import read_pictures
>>> img_data = read_pictures(['my_image.jpg'])
>>> detector = Detector()
>>> detected_faces = detector.detect_faces(frame)
>>> detected_landmarks = detector.detect_landmarks(frame, detected_faces)
>>> detector.detect_emotions(frame, detected_faces, detected_landmarks)
detect_facepose(frame, detected_faces=None, landmarks=None)

Detect facepose from image or video frame.

When used with img2pose, returns all detected poses, and facebox and landmarks are ignored. Use detect_face method in order to obtain bounding boxes corresponding to the detected poses returned by this method.

When used with pnp model, ‘facebox’ param is ignored, and the passed 2D landmarks are used to compute the head pose for the single face associated with the passed landmarks.

Parameters

  • frame (np.ndarray) – list of cv2 images

  • detected_faces (list) – (num_images, num_faces, 4) faceboxes representing faces in the list of images

  • landmarks (np.ndarray) – (num_images, num_faces, 68, 2) landmarks for the faces contained in list of images

Returns

(num_images, num_faces, [pitch, roll, yaw]) - Euler angles (in degrees) for each face within in each image

Return type

np.ndarray

Examples

>>> from feat import Detector
>>> from feat.utils import read_pictures
>>> frame = read_pictures(['my_image.jpg'])

>>> # Imgpose detector
>>> imgpose_detector = Detector(face_model='imgpose', facepose_model='img2pose')
>>> imgpose_detector.detect_facepose(frame) # one shot computation

>>> # Retina face detector
>>> retinaface_detector = Detector(face_model='retinaface', landmark_model='mobilefacenet', facepose_model='pnp')
>>> faces = retinaface_detector.detect_faces(frame)
>>> landmarks = retinaface_detector.detect_landmarks(detected_faces=faces)
>>> retinaface_detector.detect_facepose(frame=frame, landmarks=landmarks) # detect pose for all faces
detect_faces(frame)

Detect faces from image or video frame

Parameters

frame (np.ndarray) – 3d (single) or 4d (multiple) image array

Returns

list of lists with the same length as the number of frames. Each list item is a list containing the (x1, y1, x2, y2) coordinates of each detected face in that frame.

Return type

list

Examples

>>> from feat import Detector
>>> from feat.utils import read_pictures
>>> img_data = read_pictures(['my_image.jpg'])
>>> detector = Detector()
>>> detector.detect_faces(frame)
detect_image(inputFname, batch_size=5, outputFname=None, return_detection=True, singleframe4error=False)

Detects FEX from an image file.

Parameters

  • inputFname (list of str) – Path to a list of paths to image files.

  • batch_size (int, optional) – how many batches of images you want to run at one shot. Larger gives faster speed but is more memory-consuming

  • outputFname (str, optional) – Path to output file. Defaults to None.

  • return_detection (bool, optional) – whether to return a Fex object of all

  • to (concatenated detections. To save memory you can process results directly) –

  • True (a file by setting this to False and providing an outputFname; Default) –

  • singleframe4error (bool, default = False) – When set True, when exception

  • batch (instead of nullify the whole) –

  • batch –

  • img (process each) –

  • individually (in batch) –

Returns

Prediction results dataframe if outputFname is None. Returns True if outputFname is specified.

Return type

Fex

detect_landmarks(frame, detected_faces)

Detect landmarks from image or video frame

Parameters

  • frame (np.ndarray) – 3d (single) or 4d (multiple) image array

  • detected_faces (array) –

Returns

x and y landmark coordinates (1,68,2)

Return type

list

Examples

>>> from feat import Detector
>>> from feat.utils import read_pictures
>>> img_data = read_pictures(['my_image.jpg'])
>>> detector = Detector()
>>> detected_faces = detector.detect_faces(frame)
>>> detector.detect_landmarks(frame, detected_faces)
detect_video(inputFname, batch_size=5, outputFname=None, return_detection=True, skip_frames=1, singleframe4error=False)

Detects FEX from a video file.

Parameters

  • inputFname (str) – Path to video file

  • batch_size (int, optional) – how many batches of images you want to run at one shot. Larger gives faster speed but is more memory-consuming

  • outputFname (str, optional) – Path to output file. Defaults to None.

  • return_detection (bool, optional) – whether to return a Fex object of all

  • to (concatenated detections. To save memory you can process results directly) –

  • True (a file by setting this to False and providing an outputFname; Default) –

  • skip_frames (int, optional) – Number of every other frames to skip for speed or if not all frames need to be processed. Defaults to 1.

  • singleframe4error (bool, default = False) – When set True, when exception

  • batch (instead of nullify the whole) –

  • batch –

  • img (process each) –

  • individually (in batch) –

Returns

Prediction results dataframe if outputFname is None. Returns True if outputFname is specified.

Return type

dataframe

extract_face(frame, detected_faces, landmarks, size_output=112)

Extract a face in a frame with a convex hull of landmarks.

This function extracts the faces of the frame with convex hulls and masks out the rest.

Parameters

  • frame (array) – The original image]

  • detected_faces (list) – face bounding box

  • landmarks (list) – the landmark information]

  • size_output (int, optional) – [description]. Defaults to 112.

Returns

resized face as a numpy array new_landmarks: landmarks of aligned face

Return type

resized_face_np

extract_hog(frame, orientation=8, pixels_per_cell=(8, 8), cells_per_block=(2, 2), visualize=False)

Extract HOG features from a SINGLE frame.

Parameters

  • frame (array]) – Frame of image]

  • orientation (int, optional) – Orientation for HOG. Defaults to 8.

  • pixels_per_cell (tuple, optional) – Pixels per cell for HOG. Defaults to (8,8).

  • cells_per_block (tuple, optional) – Cells per block for HOG. Defaults to (2,2).

  • visualize (bool, optional) – Whether to provide the HOG image. Defaults to False.

Returns

array of HOG features, and the HOG image if visualize is True.

Return type

hog_output

process_frame(frames, input_names, counter=0, singleframe4error=False, skip_frame_rate=1, is_video_frame=False)

Function to run face detection, landmark detection, and emotion detection on a frame.

Parameters

  • frames (np.array) – batch of frames, of shape BxHxWxC (read from cv2)

  • input_names (list) – file names for each frame in the batch

  • counter (int, str, default=0) – Index used for the prediction results

  • batches (dataframe. Tracks the) –

  • singleframe4error (bool, default = False) – When exception occurs inside a

  • batch (process each img in) –

  • batch –

  • batch –

  • individually –

  • is_video_frame (bool) – Whether processing is happening over batches of

  • len (VIDEOFRAMES so) –

  • len –

  • False (Default) –

Returns

Prediction results dataframe. int: counter - the updated number of counter. Used to track the batch size and image number

Return type

feat.data.Fex (dataframe)

read_images(imgname_list)

Given a string filepath or list of filepath, load the image(s) as a numpy array

Parameters

imgname_list (list/str) – filename or list of filenames

Returns

3d or 4d numpy array

Return type

np.ndarry

feat.data module

Main Fex data class. The Fex class is a pandas DataFrame subclass that makes it easier to work with the results output from a Detector

class feat.data.Fex(*args, **kwargs)

Bases: pandas.core.frame.DataFrame

Fex is a class to represent facial expression (Fex) data

Fex class is an enhanced pandas dataframe, with extra attributes and methods to help with facial expression data analysis.

Parameters

  • filename – (str, optional) path to file

  • detector – (str, optional) name of software used to extract Fex. (Feat, FACET, OpenFace, or Affectiva)

  • sampling_freq (float, optional) – sampling rate of each row in Hz; defaults to None

  • features (pd.Dataframe, optional) – features that correspond to each Fex row

  • sessions – Unique values indicating rows associated with a specific session (e.g., trial, subject, etc).Must be a 1D array of n_samples elements; defaults to None

append(data, session_id=None, axis=0)

Append a new Fex object to an existing object

Parameters

  • data – (Fex) Fex instance to append

  • session_id – session label

  • axis – ([0,1]) Axis to append. Rows=0, Cols=1

Returns

Fex instance

property aus

Returns the Action Units data

Returns Action Unit data using the columns set in fex.au_columns.

Returns

Action Units data

Return type

DataFrame

baseline(baseline='median', normalize=None, ignore_sessions=False)

Reference a Fex object to a baseline.

Parameters

  • method – {‘median’, ‘mean’, ‘begin’, FexSeries instance}. Will subtract baseline from Fex object (e.g., mean, median). If passing a Fex object, it will treat that as the baseline.

  • normalize – (str). Can normalize results of baseline. Values can be [None, ‘db’,’pct’]; default None.

  • ignore_sessions – (bool) If True, will ignore Fex.sessions information. Otherwise, method will be applied separately to each unique session.

Returns

Fex object

calc_pspi()
clean(detrend=True, standardize=True, confounds=None, low_pass=None, high_pass=None, ensure_finite=False, ignore_sessions=False, *args, **kwargs)

Clean Time Series signal

This function wraps nilearn functionality and can filter, denoise, detrend, etc.

See http://nilearn.github.io/modules/generated/nilearn.signal.clean.html

This function can do several things on the input signals, in the following order: detrend, standardize, remove confounds, low and high-pass filter

If Fex.sessions is not None, sessions will be cleaned separately.

Parameters

  • confounds – (numpy.ndarray, str or list of Confounds timeseries) Shape must be (instant number, confound number), or just (instant number,). The number of time instants in signals and confounds must be identical (i.e. signals.shape[0] == confounds.shape[0]). If a string is provided, it is assumed to be the name of a csv file containing signals as columns, with an optional one-line header. If a list is provided, all confounds are removed from the input signal, as if all were in the same array.

  • low_pass – (float) low pass cutoff frequencies in Hz.

  • high_pass – (float) high pass cutoff frequencies in Hz.

  • detrend – (bool) If detrending should be applied on timeseries (before confound removal)

  • standardize – (bool) If True, returned signals are set to unit variance.

  • ensure_finite – (bool) If True, the non-finite values (NANs and infs) found in the data will be replaced by zeros.

  • ignore_sessions – (bool) If True, will ignore Fex.sessions information. Otherwise, method will be applied separately to each unique session.

Returns

cleaned Fex instance

decompose(algorithm='pca', axis=1, n_components=None, *args, **kwargs)

Decompose Fex instance

Parameters

  • algorithm – (str) Algorithm to perform decomposition types=[‘pca’,’ica’,’nnmf’,’fa’]

  • axis – dimension to decompose [0,1]

  • n_components – (int) number of components. If None then retain as many as possible.

Returns

a dictionary of decomposition parameters

Return type

output

property design

Returns the design data

Returns the study design information using columns in fex.design_columns.

Returns

time data

Return type

DataFrame

distance(method='euclidean', **kwargs)

Calculate distance between rows within a Fex() instance.

Parameters

method – type of distance metric (can use any scikit learn or sciypy metric)

Returns

Outputs a 2D distance matrix.

Return type

dist

downsample(target, **kwargs)
Downsample Fex columns. Relies on nltools.stats.downsample,

but ensures that returned object is a Fex object.

Parameters

  • target (float) – downsampling target, typically in samples not seconds

  • kwargs – additional inputs to nltools.stats.downsample

property emotions

Returns the emotion data

Returns emotions data using the columns set in fex.emotion_columns.

Returns

emotion data

Return type

DataFrame

extract_boft(min_freq=0.06, max_freq=0.66, bank=8, *args, **kwargs)

Extract Bag of Temporal features

Parameters

  • min_freq – maximum frequency of temporal filters

  • max_freq – minimum frequency of temporal filters

  • bank – number of temporal filter banks, filters are on exponential scale

Returns

list of Morlet wavelets with corresponding freq hzs: list of hzs for each Morlet wavelet

Return type

wavs

extract_max(ignore_sessions=False, *args, **kwargs)

Extract maximum of each feature

Parameters

ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex) maximum values for each feature

Return type

fex

extract_mean(ignore_sessions=False, *args, **kwargs)

Extract mean of each feature

Parameters

ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

mean values for each feature

Return type

Fex

extract_min(ignore_sessions=False, *args, **kwargs)

Extract minimum of each feature

Parameters

ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex) minimum values for each feature

Return type

Fex

extract_multi_wavelet(min_freq=0.06, max_freq=0.66, bank=8, *args, **kwargs)

Convolve with a bank of morlet wavelets.

Wavelets are equally spaced from min to max frequency. See extract_wavelet for more information and options.

Parameters

  • min_freq – (float) minimum frequency to extract

  • max_freq – (float) maximum frequency to extract

  • bank – (int) size of wavelet bank

  • num_cyc – (float) number of cycles for wavelet

  • mode – (str) feature to extract, e.g., [‘complex’,’filtered’,’phase’,’magnitude’,’power’]

  • ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex instance)

Return type

convolved

extract_summary(mean=True, max=True, min=True, ignore_sessions=False, *args, **kwargs)

Extract summary of multiple features

Parameters

  • mean – (bool) extract mean of features

  • max – (bool) extract max of features

  • min – (bool) extract min of features

  • ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex)

Return type

fex

extract_wavelet(freq, num_cyc=3, mode='complex', ignore_sessions=False)

Perform feature extraction by convolving with a complex morlet wavelet

Parameters

  • freq – (float) frequency to extract

  • num_cyc – (float) number of cycles for wavelet

  • mode – (str) feature to extract, e.g., ‘complex’,’filtered’,’phase’,’magnitude’,’power’]

  • ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex instance)

Return type

convolved

property facebox

Returns the facebox data

Returns the facebox data using fex.facebox_columns.

Returns

facebox data

Return type

DataFrame

property facepose

Returns the facepose data using the columns set in fex.facepose_columns

Returns

facepose data

Return type

DataFrame

property info

Print all meta data of fex

Loops through metadata set in self._metadata and prints out the information.

property input

Returns input column as string

Returns input data in the “input” column.

Returns

path to input image

Return type

string

isc(col, index='frame', columns='input', method='pearson')

[summary]

Parameters

  • col (str]) – Column name to compute the ISC for.

  • index (str, optional) – Column to be used in computing ISC. Usually this would be the column identifying the time such as the number of the frame. Defaults to “frame”.

  • columns (str, optional) – Column to be used for ISC. Usually this would be the column identifying the video or subject. Defaults to “input”.

  • method (str, optional) – Method to use for correlation pearson, kendall, or spearman. Defaults to “pearson”.

Returns

Correlation matrix with index as colmns

Return type

DataFrame

itersessions()

Iterate over Fex sessions as (session, series) pairs.

Returns

a generator that iterates over the sessions of the fex instance

Return type

it

property landmark

Returns the landmark data

Returns landmark data using the columns set in fex.landmark_columns.

Returns

landmark data

Return type

DataFrame

property landmark_x

Returns the x landmarks.

Returns the x-coordinates for facial landmarks looking for “x” in fex.landmark_columns.

Returns

x landmarks.

Return type

DataFrame

property landmark_y

Returns the y landmarks.

Returns the y-coordinates for facial landmarks looking for “y” in fex.landmark_columns.

Returns

y landmarks.

Return type

DataFrame

plot_detections(faces='landmarks', faceboxes=True, muscles=False, poses=False, gazes=False, add_titles=True, au_barplot=True, emotion_barplot=True)

Plots detection results by Feat. Can control plotting of face, AU barplot and Emotion barplot. The faces kwarg controls whether facial landmarks are draw on top of input images or whether faces are visualized using Py-Feat’s AU visualization model using detected AUs. If detection was performed on a video an faces = ‘landmarks’, only an outline of the face will be draw without loading the underlying vidoe frame to save memory.

Parameters

  • faces (str, optional) – ‘landmarks’ to draw detected landmarks or ‘aus’ to

  • model. (generate a face from AU detections using Py-Feat's AU landmark) –

  • 'landmarks'. (Defaults to) –

  • faceboxes (bool, optional) – Whether to draw the bounding box around detected

  • True. (the face. Defaults to) –

  • muscles (bool, optional) – Whether to draw muscles from AU activity. Only

  • False. (faces='landmarks'. Defaults to) –

  • poses (bool, optional) – Whether to draw facial poses. Only applies if

  • False. –

  • gazes (bool, optional) – Whether to draw gaze vectors. Only applies if faces=’aus’. Defaults to False.

  • add_titles (bool, optional) – Whether to add the file name as a title above

  • True. –

  • au_barplot (bool, optional) – Whether to include a subplot for au detections. Defaults to True.

  • emotion_barplot (bool, optional) – Whether to include a subplot for emotion detections. Defaults to True.

Returns

list of matplotlib figures

Return type

list

predict(X, y, model=<class 'sklearn.linear_model._base.LinearRegression'>, *args, **kwargs)

Predicts y from X using a sklearn model.

Predict a variable of interest y using your model of choice from X, which can be a list of columns of the Fex instance or a dataframe.

Parameters

  • X (list or DataFrame) – List of column names or dataframe to be used as features for prediction

  • y (string or array) – y values to be predicted

  • model (class, optional) – Any sklearn model. Defaults to LinearRegression.

  • args – Model arguments

  • kwargs – Model arguments

Returns

Fit model instance.

Return type

model

read_affectiva(filename=None, *args, **kwargs)

Reads facial expression detection results from Affectiva

Parameters

filename (string, optional) – Path to file. Defaults to None.

Returns

Fex

read_facet(filename=None, *args, **kwargs)

Reads facial expression detection results from FACET

Parameters

filename (string, optional) – Path to file. Defaults to None.

Returns

Fex

read_feat(filename=None, *args, **kwargs)

Reads facial expression detection results from Feat Detector

Parameters

filename (string, optional) – Path to file. Defaults to None.

Returns

Fex

read_file(*args, **kwargs)

Loads file into FEX class

This function checks the detector set in fex.detector and calls the appropriate read function that helps utilize functionalities of Feat.

Available detectors include:

FACET OpenFace Affectiva Feat

Returns

Fex class

Return type

DataFrame

read_openface(filename=None, *args, **kwargs)

Reads facial expression detection results from OpenFace

Parameters

filename (string, optional) – Path to file. Defaults to None.

Returns

Fex

rectification(std=3)

Removes time points when the face position moved more than N standard deviations from the mean.

Parameters

std (default 3) – standard deviation from mean to remove outlier face locations

Returns

cleaned FEX object

Return type

data

regress(X, y, fit_intercept=True, *args, **kwargs)

Regress using nltools.stats.regress.

fMRI-like regression to predict Fex activity (y) from set of regressors (X).

Parameters

  • X (list or str) – Independent variable to predict.

  • y (list or str) – Dependent variable to be predicted.

  • fit_intercept (bool) – Whether to add intercept before fitting. Defaults to True.

Returns

betas, t-stats, p-values, df, residuals

property time

Returns the time data

Returns the time information using fex.time_columns.

Returns

time data

Return type

DataFrame

ttest_1samp(popmean=0, threshold_dict=None)

Conducts 1 sample ttest.

Uses scipy.stats.ttest_1samp to conduct 1 sample ttest

Parameters

  • popmean (int, optional) – Population mean to test against. Defaults to 0.

  • threshold_dict ([type], optional) – Dictonary for thresholding. Defaults to None. [NOT IMPLEMENTED]

Returns

t-statistics and p-values

Return type

t, p

ttest_ind(col, sessions, threshold_dict=None)

Conducts 2 sample ttest.

Uses scipy.stats.ttest_ind to conduct 2 sample ttest on column col between sessions.

Parameters

  • col (str) – Column names to compare in a t-test between sessions

  • session_names (tuple) – tuple of session names stored in Fex.sessions.

  • threshold_dict ([type], optional) – Dictonary for thresholding. Defaults to None. [NOT IMPLEMENTED]

Returns

t-statistics and p-values

Return type

t, p

upsample(target, target_type='hz', **kwargs)
Upsample Fex columns. Relies on nltools.stats.upsample,

but ensures that returned object is a Fex object.

Parameters

  • target (float) – upsampling target, default ‘hz’ (also ‘samples’, ‘seconds’)

  • kwargs – additional inputs to nltools.stats.upsample

class feat.data.FexSeries(*args, **kwargs)

Bases: pandas.core.series.Series

This is a sub-class of pandas series. While not having additional methods of it’s own required to retain normal slicing functionality for the Fex class, i.e. how slicing is typically handled in pandas. All methods should be called on Fex below.

property aus

Returns the Action Units data

Returns

Action Units data

Return type

DataFrame

property design

Returns the design data

Returns

time data

Return type

DataFrame

property emotions

Returns the emotion data

Returns

emotion data

Return type

DataFrame

property facebox

Returns the facebox data

Returns

facebox data

Return type

DataFrame

property facepose

Returns the facepose data

Returns

facepose data

Return type

DataFrame

property info

Print class meta data.

property input

Returns input column as string

Returns

path to input image

Return type

string

property landmark

Returns the landmark data

Returns

landmark data

Return type

DataFrame

property landmark_x

Returns the x landmarks.

Returns

x landmarks.

Return type

DataFrame

property landmark_y

Returns the y landmarks.

Returns

y landmarks.

Return type

DataFrame

plot_detections(*args, **kwargs)

Alias for Fex.plot_detections

property time

Returns the time data

Returns

time data

Return type

DataFrame

class feat.data.Fextractor

Bases: object

Fextractor is a class that extracts and merges features from a Fex instance in preparation for data analysis.

boft(fex_object, min_freq=0.06, max_freq=0.66, bank=8, *args, **kwargs)

Extract Bag of Temporal features

Parameters

  • fex_object – (Fex) Fex instance to extract features from.

  • min_freq – maximum frequency of temporal filters

  • max_freq – minimum frequency of temporal filters

  • bank – number of temporal filter banks, filters are on exponential scale

Returns

list of Morlet wavelets with corresponding freq hzs: list of hzs for each Morlet wavelet

Return type

wavs

max(fex_object, ignore_sessions=False, *args, **kwargs)

Extract maximum of each feature

Parameters

  • fex_object – (Fex) Fex instance to extract features from.

  • ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex) maximum values for each feature

Return type

Fex

mean(fex_object, ignore_sessions=False, *args, **kwargs)

Extract mean of each feature

Parameters

  • fex_object – (Fex) Fex instance to extract features from.

  • ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

mean values for each feature

Return type

Fex

merge(out_format='long')

Merge all extracted features to a single dataframe

Parameters

format – (str) Output format of merged data. Can be set to ‘long’ or ‘wide’. Defaults to long.

Returns

(DataFrame) DataFrame containing merged features extracted from a Fex instance.

Return type

merged

min(fex_object, ignore_sessions=False, *args, **kwargs)

Extract minimum of each feature

Parameters

  • fex_object – (Fex) Fex instance to extract features from.

  • ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex) minimum values for each feature

Return type

Fex

multi_wavelet(fex_object, min_freq=0.06, max_freq=0.66, bank=8, *args, **kwargs)

Convolve with a bank of morlet wavelets.

Wavelets are equally spaced from min to max frequency. See extract_wavelet for more information and options.

Parameters

  • fex_object – (Fex) Fex instance to extract features from.

  • min_freq – (float) minimum frequency to extract

  • max_freq – (float) maximum frequency to extract

  • bank – (int) size of wavelet bank

  • num_cyc – (float) number of cycles for wavelet

  • mode – (str) feature to extract, e.g., [‘complex’,’filtered’,’phase’,’magnitude’,’power’]

  • ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex instance)

Return type

convolved

summary(fex_object, mean=False, max=False, min=False, ignore_sessions=False, *args, **kwargs)

Extract summary of multiple features

Parameters

  • fex_object – (Fex) Fex instance to extract features from.

  • mean – (bool) extract mean of features

  • max – (bool) extract max of features

  • min – (bool) extract min of features

  • ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex)

Return type

fex

wavelet(fex_object, freq, num_cyc=3, mode='complex', ignore_sessions=False)

Perform feature extraction by convolving with a complex morlet wavelet

Parameters

  • fex_object – (Fex) Fex instance to extract features from.

  • freq – (float) frequency to extract

  • num_cyc – (float) number of cycles for wavelet

  • mode – (str) feature to extract, e.g., [‘complex’,’filtered’,’phase’,’magnitude’,’power’]

  • ignore_sessions – (bool) ignore sessions or extract separately by sessions if available.

Returns

(Fex instance)

Return type

convolved

feat.plotting module

Helper functions for plotting

feat.plotting.animate_face(AU=None, start=None, end=None, save=None, include_reverse=True, **kwargs)

Create a matplotlib animation interpolating between a starting and ending face. Can either work like plot_face by taking an array of AU intensities for start and end, or by animating a single AU using the AU keyword argument and setting start and end to a scalar value

Parameters

  • AU (str/int, optional) – action unit id (e.g. 12 or ‘AU12’). Defaults to None.

  • start (float/np.ndarray, optional) – AU intensity to start at. Defaults to None.

  • end (float/np.ndarray, optional) – AU intensity(s) to end at. We don’t recommend

  • None. (going beyond 3. Defaults to) –

  • save (str, optional) – file to save animation to. Defaults to None.

  • include_reverse (bool, optional) – Whether to also reverse the animation, i.e.

  • True. (start -> end -> start. Defaults to) –

  • title (str, optional) – plot title. Defaults to None.

  • fps (int, optional) – frame-rate; Defaults to 15fps

  • duration (float, optional) – length of animation in seconds. Defaults to 0.5

  • padding (float, optional) – additional time to wait in seconds on the first and

  • animation. (last frame of the animation. Useful when you plan to loop the) –

  • 0.25 (Defaults to) –

  • interp_func (callable, optional) – interpolation function that takes a start and

  • values (end keyword argument and returns a function that will be applied to) –

  • np.linspace (0, 1, num_frames) –

  • https – //github.com/semitable/easing-functions for other options.

Returns

matplotlib Animation

feat.plotting.draw_lineface(currx, curry, ax=None, color='k', linestyle='-', linewidth=1, gaze=None, *args, **kwargs)

Plot Line Face

Parameters

  • currx – vector (len(68)) of x coordinates

  • curry – vector (len(68)) of y coordinates

  • ax – matplotlib axis to add

  • color – matplotlib line color

  • linestyle – matplotlib linestyle

  • linewidth – matplotlib linewidth

  • gaze – array (len(4)) of gaze vectors (fifth value is whether to draw vectors)

feat.plotting.draw_muscles(currx, curry, au=None, ax=None, *args, **kwargs)

Draw Muscles

Parameters

  • currx – vector (len(68)) of x coordinates

  • curry – vector (len(68)) of y coordinates

  • ax – matplotlib axis to add

feat.plotting.draw_vectorfield(reference, target, color='r', scale=1, width=0.007, ax=None, *args, **kwargs)

Draw vectorfield from reference to target

Parameters

  • reference – reference landmarks (2,68)

  • target – target landmarks (2,68)

  • ax – matplotlib axis instance

  • au – vector of action units (len(17))

feat.plotting.get_heat(muscle, au, log)

Function to create heatmap from au vector

Parameters

  • muscle (string) – string representation of a muscle

  • au (list) – vector of action units

  • log (boolean) – whether the action unit values are on a log scale

Returns

color of muscle according to its au value

feat.plotting.imshow(obj, figsize=None, aspect='equal')

Convenience wrapper function around matplotlib imshow that creates figure and axis boilerplate for single image plotting

Parameters

  • obj (str/Path/PIL.Imag) – string or Path to image file or pre-loaded PIL.Image instance

  • figsize (tuple, optional) – matplotlib figure size. Defaults to None.

  • aspect (str, optional) – passed to matplotlib imshow. Defaults to “equal”.

feat.plotting.interpolate_aus(start, end, num_frames, interp_func=None, num_padding_frames=None, include_reverse=True)

Helper function to interpolate between starting and ending AU values using non-linear easing functions

Parameters

  • start (np.ndarray) – array of starting intensities

  • end (np.ndarray) – array of ending intensities

  • num_frames (int) – number of frames to interpolate over

  • interp_func (callable, optional) – easing function. Defaults to None.

  • num_padding_frames (int, optional) – number of additional freeze frames to add

  • None. (before the first frame and after the last frame. Defaults to) –

  • include_reverse (bool, optional) – return the reverse interpolation appended to

  • True. (the end of the interpolation. Useful for animating start -> end -> start. Defaults to) –

Returns

frames x au 2d array

Return type

np.ndarray

feat.plotting.plot_face(model=None, au=None, vectorfield=None, muscles=None, ax=None, feature_range=False, color='k', linewidth=1, linestyle='-', gaze=None, muscle_scaler=None, *args, **kwargs)

Core face plotting function

Parameters

  • model (landmark AU) – sklearn PLSRegression instance; Default None which uses Py-Feat’s

  • model –

  • au – vector of action units (same length as model.n_components)

  • vectorfield – (dict) {‘target’:target_array,’reference’:reference_array}

  • muscles – (dict) {‘muscle’: color}

  • ax – matplotlib axis handle

  • (tuple (feature_range) – None): If a tuple with (min, max), scale input AU intensities to (min, max) before prediction.

  • default – None): If a tuple with (min, max), scale input AU intensities to (min, max) before prediction.

  • color – matplotlib color

  • linewidth – matplotlib linewidth

  • linestyle – matplotlib linestyle

  • gaze – array of gaze vectors (len(4))

Returns

plot handle

Return type

ax

feat.plotting.predict(au, model=None, feature_range=None)

Helper function to predict landmarks from au given a sklearn model

Parameters

  • au – vector of action unit intensities

  • model – sklearn pls object (uses pretrained model by default)

  • (tuple (feature_range) – None): If a tuple with (min, max), scale input AU intensities to (min, max) before prediction.

  • default – None): If a tuple with (min, max), scale input AU intensities to (min, max) before prediction.

Returns

Array of landmarks (2,68)

Return type

landmarks

feat.utils module

Feat utility and helper functions. Some of these functions can be used directly, while other underly the functionality of various Detector or Fex methods.

feat.utils.download_url(*args, **kwargs)

By default just call download_url from torch vision, but we pass a verbose = False keyword argument, then call download_url with a special context manager than supresses the print messages

feat.utils.get_resource_path()

Get path to feat resource directory.

feat.utils.get_test_data_path()

Get path to feat test data directory.

feat.utils.load_h5(file_name='pyfeat_aus_to_landmarks', prefer_joblib_if_version_match=True)

Load the h5 PLS model for plotting. Will try using joblib if python and sklearn major and minor versions match those the model was trained with (3.8.x and 1.0.x respectively), otherwise will reconstruct the model object using h5 data.

Parameters

  • file_name (str, optional) – Specify model to load.. Defaults to ‘blue.h5’.

  • prefer_joblib_if_version_match (bool, optional) – If the sklearn and python major.minor versions

  • from (match then return the pickled PLSRegression object. Otherwise build it) –

  • True (scratch using .h5 data. Default) –

Returns

PLS model

Return type

model

feat.utils.read_affectiva(affectivafile, orig_cols=False)

This function reads in affectiva file processed through the https://github.com/cosanlab/affectiva-api-app.

Parameters

  • affectivafile – file to read

  • orig_cols – If True, convert original colnames to FACS names

Returns

Fex of processed facial expressions

feat.utils.read_facet(facetfile, features=None, raw=False, sampling_freq=None)

This function reads in an iMotions-FACET exported facial expression file.

Parameters

  • facetfile – iMotions-FACET file. Files from iMotions 5, 6, and 7 have been tested and supported

  • features – If a list of iMotion-FACET column names are passed, those are returned. Otherwise, default columns are returned in the following format:[‘Timestamp’,’FaceRectX’,’FaceRectY’,’FaceRectWidth’,’FaceRectHeight’, ‘Joy’,’Anger’,’Surprise’,’Fear’,’Contempt’, ‘Disgust’,’Sadness’,’Confusion’,’Frustration’, ‘Neutral’,’Positive’,’Negative’,’AU1’,’AU2’, ‘AU4’,’AU5’,’AU6’,’AU7’,’AU9’,’AU10’, ‘AU12’,’AU14’,’AU15’,’AU17’,’AU18’,’AU20’, ‘AU23’,’AU24’,’AU25’,’AU26’,’AU28’,’AU43’, ‘Yaw’, ‘Pitch’, ‘Roll’]. Note that these column names are different from the original files which has ‘ Evidence’, ‘ Degrees’ appended to each column.

  • raw (default=False) – Set to True to return all columns without processing.

  • sampling_freq – sampling frequency to pass to Fex

Returns

dataframe of processed facial expressions

feat.utils.read_openface(openfacefile, features=None)

This function reads in an OpenFace exported facial expression file. :param features: If a list of column names are passed, those are returned. Otherwise, default returns the following features: :param [‘frame’: :param ‘timestamp’: :param ‘confidence’: :param ‘success’: :param ‘gaze_0_x’: :param :

‘gaze_0_y’, ‘gaze_0_z’, ‘gaze_1_x’, ‘gaze_1_y’, ‘gaze_1_z’, ‘pose_Tx’, ‘pose_Ty’, ‘pose_Tz’, ‘pose_Rx’, ‘pose_Ry’, ‘pose_Rz’, ‘x_0’, ‘x_1’, ‘x_2’, ‘x_3’, ‘x_4’, ‘x_5’, ‘x_6’, ‘x_7’, ‘x_8’, ‘x_9’, ‘x_10’, ‘x_11’, ‘x_12’, ‘x_13’, ‘x_14’, ‘x_15’, ‘x_16’, ‘x_17’, ‘x_18’, ‘x_19’, ‘x_20’, ‘x_21’, ‘x_22’, ‘x_23’, ‘x_24’, ‘x_25’, ‘x_26’, ‘x_27’, ‘x_28’, ‘x_29’, ‘x_30’, ‘x_31’, ‘x_32’, ‘x_33’, ‘x_34’, ‘x_35’, ‘x_36’, ‘x_37’, ‘x_38’, ‘x_39’, ‘x_40’, ‘x_41’, ‘x_42’, ‘x_43’, ‘x_44’, ‘x_45’, ‘x_46’, ‘x_47’, ‘x_48’, ‘x_49’, ‘x_50’, ‘x_51’, ‘x_52’, ‘x_53’, ‘x_54’, ‘x_55’, ‘x_56’, ‘x_57’, ‘x_58’, ‘x_59’, ‘x_60’, ‘x_61’, ‘x_62’, ‘x_63’, ‘x_64’, ‘x_65’, ‘x_66’, ‘x_67’, ‘y_0’, ‘y_1’, ‘y_2’, ‘y_3’, ‘y_4’, ‘y_5’, ‘y_6’, ‘y_7’, ‘y_8’, ‘y_9’, ‘y_10’, ‘y_11’, ‘y_12’, ‘y_13’, ‘y_14’, ‘y_15’, ‘y_16’, ‘y_17’, ‘y_18’, ‘y_19’, ‘y_20’, ‘y_21’, ‘y_22’, ‘y_23’, ‘y_24’, ‘y_25’, ‘y_26’, ‘y_27’, ‘y_28’, ‘y_29’, ‘y_30’, ‘y_31’, ‘y_32’, ‘y_33’, ‘y_34’, ‘y_35’, ‘y_36’, ‘y_37’, ‘y_38’, ‘y_39’, ‘y_40’, ‘y_41’, ‘y_42’, ‘y_43’, ‘y_44’, ‘y_45’, ‘y_46’, ‘y_47’, ‘y_48’, ‘y_49’, ‘y_50’, ‘y_51’, ‘y_52’, ‘y_53’, ‘y_54’, ‘y_55’, ‘y_56’, ‘y_57’, ‘y_58’, ‘y_59’, ‘y_60’, ‘y_61’, ‘y_62’, ‘y_63’, ‘y_64’, ‘y_65’, ‘y_66’, ‘y_67’, ‘X_0’, ‘X_1’, ‘X_2’, ‘X_3’, ‘X_4’, ‘X_5’, ‘X_6’, ‘X_7’, ‘X_8’, ‘X_9’, ‘X_10’, ‘X_11’, ‘X_12’, ‘X_13’, ‘X_14’, ‘X_15’, ‘X_16’, ‘X_17’, ‘X_18’, ‘X_19’, ‘X_20’, ‘X_21’, ‘X_22’, ‘X_23’, ‘X_24’, ‘X_25’, ‘X_26’, ‘X_27’, ‘X_28’, ‘X_29’, ‘X_30’, ‘X_31’, ‘X_32’, ‘X_33’, ‘X_34’, ‘X_35’, ‘X_36’, ‘X_37’, ‘X_38’, ‘X_39’, ‘X_40’, ‘X_41’, ‘X_42’, ‘X_43’, ‘X_44’, ‘X_45’, ‘X_46’, ‘X_47’, ‘X_48’, ‘X_49’, ‘X_50’, ‘X_51’, ‘X_52’, ‘X_53’, ‘X_54’, ‘X_55’, ‘X_56’, ‘X_57’, ‘X_58’, ‘X_59’, ‘X_60’, ‘X_61’, ‘X_62’, ‘X_63’, ‘X_64’, ‘X_65’, ‘X_66’, ‘X_67’, ‘Y_0’, ‘Y_1’, ‘Y_2’, ‘Y_3’, ‘Y_4’, ‘Y_5’, ‘Y_6’, ‘Y_7’, ‘Y_8’, ‘Y_9’, ‘Y_10’, ‘Y_11’, ‘Y_12’, ‘Y_13’, ‘Y_14’, ‘Y_15’, ‘Y_16’, ‘Y_17’, ‘Y_18’, ‘Y_19’, ‘Y_20’, ‘Y_21’, ‘Y_22’, ‘Y_23’, ‘Y_24’, ‘Y_25’, ‘Y_26’, ‘Y_27’, ‘Y_28’, ‘Y_29’, ‘Y_30’, ‘Y_31’, ‘Y_32’, ‘Y_33’, ‘Y_34’, ‘Y_35’, ‘Y_36’, ‘Y_37’, ‘Y_38’, ‘Y_39’, ‘Y_40’, ‘Y_41’, ‘Y_42’, ‘Y_43’, ‘Y_44’, ‘Y_45’, ‘Y_46’, ‘Y_47’, ‘Y_48’, ‘Y_49’, ‘Y_50’, ‘Y_51’, ‘Y_52’, ‘Y_53’, ‘Y_54’, ‘Y_55’, ‘Y_56’, ‘Y_57’, ‘Y_58’, ‘Y_59’, ‘Y_60’, ‘Y_61’, ‘Y_62’, ‘Y_63’, ‘Y_64’, ‘Y_65’, ‘Y_66’, ‘Y_67’, ‘Z_0’, ‘Z_1’, ‘Z_2’, ‘Z_3’, ‘Z_4’, ‘Z_5’, ‘Z_6’, ‘Z_7’, ‘Z_8’, ‘Z_9’, ‘Z_10’, ‘Z_11’, ‘Z_12’, ‘Z_13’, ‘Z_14’, ‘Z_15’, ‘Z_16’, ‘Z_17’, ‘Z_18’, ‘Z_19’, ‘Z_20’, ‘Z_21’, ‘Z_22’, ‘Z_23’, ‘Z_24’, ‘Z_25’, ‘Z_26’, ‘Z_27’, ‘Z_28’, ‘Z_29’, ‘Z_30’, ‘Z_31’, ‘Z_32’, ‘Z_33’, ‘Z_34’, ‘Z_35’, ‘Z_36’, ‘Z_37’, ‘Z_38’, ‘Z_39’, ‘Z_40’, ‘Z_41’, ‘Z_42’, ‘Z_43’, ‘Z_44’, ‘Z_45’, ‘Z_46’, ‘Z_47’, ‘Z_48’, ‘Z_49’, ‘Z_50’, ‘Z_51’, ‘Z_52’, ‘Z_53’, ‘Z_54’, ‘Z_55’, ‘Z_56’, ‘Z_57’, ‘Z_58’, ‘Z_59’, ‘Z_60’, ‘Z_61’, ‘Z_62’, ‘Z_63’, ‘Z_64’, ‘Z_65’, ‘Z_66’, ‘Z_67’, ‘p_scale’, ‘p_rx’, ‘p_ry’, ‘p_rz’, ‘p_tx’, ‘p_ty’, ‘p_0’, ‘p_1’, ‘p_2’, ‘p_3’, ‘p_4’, ‘p_5’, ‘p_6’, ‘p_7’, ‘p_8’, ‘p_9’, ‘p_10’, ‘p_11’, ‘p_12’, ‘p_13’, ‘p_14’, ‘p_15’, ‘p_16’, ‘p_17’, ‘p_18’, ‘p_19’, ‘p_20’, ‘p_21’, ‘p_22’, ‘p_23’, ‘p_24’, ‘p_25’, ‘p_26’, ‘p_27’, ‘p_28’, ‘p_29’, ‘p_30’, ‘p_31’, ‘p_32’, ‘p_33’, ‘AU01_r’, ‘AU02_r’, ‘AU04_r’, ‘AU05_r’, ‘AU06_r’, ‘AU07_r’, ‘AU09_r’, ‘AU10_r’, ‘AU12_r’, ‘AU14_r’, ‘AU15_r’, ‘AU17_r’, ‘AU20_r’, ‘AU23_r’, ‘AU25_r’, ‘AU26_r’, ‘AU45_r’, ‘AU01_c’, ‘AU02_c’, ‘AU04_c’, ‘AU05_c’, ‘AU06_c’, ‘AU07_c’, ‘AU09_c’, ‘AU10_c’, ‘AU12_c’, ‘AU14_c’, ‘AU15_c’, ‘AU17_c’, ‘AU20_c’, ‘AU23_c’, ‘AU25_c’, ‘AU26_c’, ‘AU28_c’, ‘AU45_c’]

Returns

dataframe of processed facial expressions

feat.utils.read_pictures(imgname_list)

Reads in a list of pictures and concatenate these pictures into batches of images.

Parameters

imgname_list (list of string) – a list of filenames for the facial pictures

Returns

np array of shape BxHxWxC

Return type

img_batch_arr (np.array)

feat.utils.registration(face_lms, neutral=array([[37.51499407, 118.99554304], [38.34746726, 135.93119299], [40.77550103, 152.83280453], [44.10928582, 169.12794022], [49.98283172, 184.53328584], [59.18894827, 198.0161361], [70.41509055, 209.2829929], [83.65962788, 217.82577978], [98.67478614, 220.00636722], [113.36502269, 217.35622274], [126.09720342, 208.6155414], [137.37278217, 197.26636201], [146.15109522, 183.95054535], [151.72032547, 168.70328048], [154.90171534, 152.54959547], [157.01705756, 136.07919401], [157.81240022, 119.28714732], [45.87342276, 109.05187535], [53.83702202, 101.43275043], [65.6123153, 99.44649503], [77.49003982, 101.34627038], [88.31833069, 105.66229287], [108.80512998, 105.18583249], [120.18051884, 100.8485088], [131.67122653, 99.22426247], [142.80408737, 101.39810664], [150.09271076, 108.74640334], [98.93955017, 117.16643104], [99.0113979, 128.44882091], [99.09059392, 139.7133518], [99.22411613, 151.32196735], [85.97238779, 158.19140086], [92.20644686, 160.61659752], [98.67862474, 162.56437315], [105.26853264, 160.62509056], [111.14227857, 158.32687793], [59.22833204, 118.63189571], [66.08746862, 114.39263502], [74.66886627, 114.59919006], [81.80683311, 120.00819189], [74.3442616, 121.70551759], [65.72377694, 121.82223252], [114.75228899, 119.90654628], [122.2983238, 114.26349216], [130.61954433, 114.38399043], [137.03708639, 118.48489575], [131.21518765, 121.51217889], [122.97461038, 121.56526096], [75.39827955, 179.40706409], [84.55991401, 176.2145797], [92.90235587, 174.4243212], [98.56534032, 176.06536597], [104.97777373, 174.45766844], [113.11257495, 176.39970964], [121.19973609, 179.19790185], [113.16310624, 185.69051009], [105.26365305, 188.31443911], [98.41771871, 188.96563941], [92.22402827, 188.38538897], [84.05109731, 185.74954658], [79.18422925, 179.80657222], [92.71723171, 179.52017819], [98.52973445, 180.16303655], [105.05932173, 179.42368921], [117.43706438, 179.71092599], [104.90869095, 180.32984592], [98.35933953, 181.1598177], [92.49485175, 180.48994809]]), method='fullface')

Register faces to a neutral face.

Affine registration of face landmarks to neutral face.

Parameters

  • face_lms (array) – face landmarks to register with shape (n,136). Columns 0~67 are x coordinates and 68~136 are y coordinates

  • neutral (array) – target neutral face array that face_lm will be registered

  • method (str or list) – If string, register to all landmarks (‘fullface’, default), or inner parts of face nose,mouth,eyes, and brows (‘inner’). If list, pass landmarks to register to e.g. [27, 28, 29, 30, 36, 39, 42, 45]

Returns

registered landmarks in shape (n,136)

Return type

registered_lms

feat.utils.softmax(x)

Softmax function to change log likelihood evidence values to probabilities. Use with Evidence values from FACET.

Parameters

x – value to softmax

feat.utils.validate_input(inputFname)

Given a string filename or list containing string files names, ensures that the file(s) exist. Always returns a non-nested list, potentionally containing a single element.

Parameters

inputFname (str or list) – file name(s)

Raises

FileNotFoundError – if any file name(s) don’t exist

Returns

list of file names (even if input was a str)

Return type

list

feat.pretrained module

Helper functions specifically for working with included pre-trained models

feat.pretrained.fetch_model(model_type, model_name)

Fetch a pre-trained model class constructor. Used by detector init

feat.pretrained.get_pretrained_models(face_model, landmark_model, au_model, emotion_model, facepose_model, verbose)

Helper function that validates the request model names and downloads them if necessary using the URLs in the included JSON file. User by detector init

previous

Extracting labels and landmarks

next

General contributions guidelines