5. Running a full analysis¶

Written by Jin Hyun Cheong and Eshin Jolly

In this tutorial we’ll perform a real analysis on the dataset from “A Data-Driven Characterisation Of Natural Facial Expressions When Giving Good And Bad News” by Watson & Johnston 2020. You can try it out interactively in Google Collab:

In this analysis, we’ll explore any differences in facial expressions people made when receiving good or bad news (see the original paper for more details). The full dataset is available on OSF, so we’ll first download the videos and their metadata. Then, we’ll load up some pre-trained emotion and action-unit (AU) detectors and use them to extract AU intensity from the videos in the dataset. Finally, we’ll demonstrate how easy it is to perform several different kinds of comparisons acoss experimental conditions (good vs bad) using the Fex data class.

# Uncomment the line below and run this only if you're using Google Collab
# !pip install -q py-feat

5.1 Download the data¶

First we’ll need to download all 20 video files and their corresponding attributes CSV files from OSF. The next cell should run quickly on Google Collab, but will depend on your own internet conection if you’re executing this notebook locally. You can rerun this cell in case the download fails for any reason, as it should skip downloading existing files:

import os
import glob
import subprocess
import numpy as np
import pandas as pd
import seaborn as sns
sns.set_context("talk")

files_to_download = {
    "4c5mb": 'clip_attrs.csv',
    "n6rt3": '001.mp4',
    "3gh8v": '002.mp4',
    "twqxs": '003.mp4',
    "nc7d9": '004.mp4',
    "nrwcm": '005.mp4',
    "2rk9c": '006.mp4',
    "mxkzq": '007.mp4',
    "c2na7": '008.mp4',
    "wj7zy": '009.mp4',
    "mxywn": '010.mp4',
    "6bn3g": '011.mp4',
    "jkwsp": '012.mp4',
    "54gtv": '013.mp4',
    "c3hpm": '014.mp4',
    "utdqj": '015.mp4',
    "hpw4a": '016.mp4',
    "94swe": '017.mp4',
    "qte5y": '018.mp4',
    "aykvu": '019.mp4',
    "3d5ry": '020.mp4',
}

for fid, fname in files_to_download.items():
    if not os.path.exists(fname):
        subprocess.run(f"wget -O {fname} --content-disposition https://osf.io/{fid}/download".split())
    else:
        print(f'Already downloaded: {fname}')

clip_attrs = pd.read_csv("clip_attrs.csv")
videos = np.sort(glob.glob("*.mp4"))
print(videos)

Already downloaded: clip_attrs.csv
Already downloaded: 001.mp4
Already downloaded: 002.mp4
Already downloaded: 003.mp4
Already downloaded: 004.mp4
Already downloaded: 005.mp4
Already downloaded: 006.mp4
Already downloaded: 007.mp4
Already downloaded: 008.mp4
Already downloaded: 009.mp4
Already downloaded: 010.mp4
Already downloaded: 011.mp4
Already downloaded: 012.mp4
Already downloaded: 013.mp4
Already downloaded: 014.mp4
Already downloaded: 015.mp4
Already downloaded: 016.mp4
Already downloaded: 017.mp4
Already downloaded: 018.mp4
Already downloaded: 019.mp4
Already downloaded: 020.mp4
['001.mp4' '002.mp4' '003.mp4' '004.mp4' '005.mp4' '006.mp4' '007.mp4'
 '008.mp4' '009.mp4' '010.mp4' '011.mp4' '012.mp4' '013.mp4' '014.mp4'
 '015.mp4' '016.mp4' '017.mp4' '018.mp4' '019.mp4' '020.mp4']

5.2 Process videos with a Detector¶

Now we’ll initialize a new Detector and process each frame of each video, saving the results to csv files named after the video.

from feat import Detector

detector = Detector(au_model = "rf", emotion_model = "resmasknet")

for video in videos: 
    out_name = video.replace(".mp4", ".csv")
    if os.path.exists(out_name):
        print(f"Already processed: {video}")
    else:
        print(f"Processing: {video}")
        detector.detect_video(inputFname = video, outputFname = out_name)

Already processed: 001.mp4
Already processed: 002.mp4
Already processed: 003.mp4
Already processed: 004.mp4
Already processed: 005.mp4
Already processed: 006.mp4
Already processed: 007.mp4
Already processed: 008.mp4
Already processed: 009.mp4
Already processed: 010.mp4
Already processed: 011.mp4
Already processed: 012.mp4
Already processed: 013.mp4
Already processed: 014.mp4
Already processed: 015.mp4
Already processed: 016.mp4
Already processed: 017.mp4
Already processed: 018.mp4
Already processed: 019.mp4
Already processed: 020.mp4

Then we can use read_feat to load each CSV file and concatenate them together:

from feat.utils import read_feat
import pandas as pd

for ix ,video in enumerate(videos):
    out_name = video.replace(".mp4", ".csv")
    if ix == 0: 
        fex = read_feat(out_name)
    else:
        fex = pd.concat([fex, read_feat(out_name)])

fex = fex.dropna()
fex.head()

	frame	FaceRectX	FaceRectY	FaceRectWidth	FaceRectHeight	FaceScore	x_0	x_1	x_2	x_3	...	Roll	Yaw	anger	disgust	fear	happiness	sadness	surprise	neutral	input
0	0	353.55045	284.39127	391.98178	528.90220	0.999796	361.880771	364.289369	370.594672	380.223391	...	-2.339310	1.250120	0.000156	0.000084	0.000831	0.962823	0.001139	0.029125	0.005842	001.mp4
1	1	353.57070	284.55023	391.95688	528.65340	0.999795	361.880771	364.289369	370.594672	380.223391	...	-2.339310	1.250120	0.000156	0.000084	0.000831	0.962823	0.001139	0.029125	0.005842	001.mp4
2	2	356.05103	284.03060	390.63580	527.54846	0.999777	361.573799	363.808128	370.042310	379.564765	...	-2.135026	1.276191	0.000164	0.000071	0.001515	0.851698	0.000759	0.141473	0.004320	001.mp4
3	3	372.93625	282.33966	387.16977	529.44080	0.999792	361.860681	364.592245	371.379240	381.879548	...	-2.460083	1.462515	0.000224	0.000105	0.001229	0.867401	0.000670	0.127383	0.002988	001.mp4
4	4	360.22266	280.82220	387.91090	522.95844	0.999802	366.604045	369.182709	375.583051	385.617726	...	-2.483959	1.655841	0.000161	0.000186	0.000778	0.964715	0.001320	0.029267	0.003572	001.mp4

5 rows × 173 columns

Lastly we can match the processed file names to the text that participants heard in the experiment using the attributes file:

# Load in attributes
clip_attrs = pd.read_csv("clip_attrs.csv")

# Add in file names and rename conditions
clip_attrs = clip_attrs.assign(
    input=clip_attrs.clipN.apply(lambda x: str(x).zfill(3) + ".mp4"),
    condition=clip_attrs["class"].replace({"gn": "goodNews", "ists": "badNews"}),
)

clip_attrs.head()

	clipN	class	phraseN	phrase_txt	input	condition
0	1	gn	1	your loan has been approved	001.mp4	goodNews
1	2	gn	2	you've got the job	002.mp4	goodNews
2	3	gn	3	the vendor has accepted your offer	003.mp4	goodNews
3	4	gn	4	your tests have come back clear	004.mp4	goodNews
4	5	gn	5	your application has been accepted	005.mp4	goodNews

5.3 Summary statistics and feature extraction¶

Fex data classes have a special attribute called .sessions that allow you to more easily compute summary statistics while respecting the grouping of an experimental setup. For example, if we set .sessions to the name of each video, then we can easily calculate mean AU intensity separately for each video using .extract_mean().

# Set sessions to the input filenames
fex.sessions = fex['input']

# Now extracting the mean is grouped by video 
average_au_intensity_per_video = fex.extract_mean()
average_au_intensity_per_video.head()

/Users/Esh/Documents/pypackages/py-feat/feat/data.py:1098: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.
  feats.append(pd.Series(v.mean(), name=k))

	mean_frame	mean_FaceRectX	mean_FaceRectY	mean_FaceRectWidth	mean_FaceRectHeight	mean_FaceScore	mean_x_0	mean_x_1	mean_x_2	mean_x_3	...	mean_Pitch	mean_Roll	mean_Yaw	mean_anger	mean_disgust	mean_fear	mean_happiness	mean_sadness	mean_surprise	mean_neutral
001.mp4	15.333333	363.440811	276.011551	389.400380	531.317063	0.999798	363.315102	366.280346	373.149525	383.261043	...	0.177313	-2.114715	1.180729	0.000413	0.000135	0.004354	0.807167	0.002137	0.180481	0.005313
002.mp4	13.500000	357.046632	271.387664	388.206440	534.764812	0.999801	356.668935	359.150546	365.668823	375.458671	...	0.283955	-1.140242	-0.137504	0.000334	0.000193	0.003266	0.689485	0.003445	0.296081	0.007197
003.mp4	23.000000	346.714941	270.177871	386.564368	536.648572	0.999812	350.511296	351.734623	356.819917	365.767994	...	0.338764	-0.764419	-0.115190	0.000173	0.000066	0.003915	0.390239	0.000757	0.602745	0.002105
004.mp4	22.000000	334.102018	280.312059	386.093014	530.022378	0.999812	335.403217	338.709198	345.324274	355.286428	...	1.916862	-2.290477	0.568985	0.000567	0.000201	0.005668	0.657875	0.004872	0.324604	0.006213
005.mp4	22.000000	320.158563	295.608988	386.956427	528.865910	0.999818	321.261506	324.980402	332.101778	342.213722	...	1.006443	-2.907811	0.906388	0.000209	0.000059	0.002261	0.870938	0.001409	0.121527	0.003597

5 rows × 172 columns

However, you can set .sessions to any grouping you desire. For the current analysis, setting .sessions equal to the condition each frame belonged to, allows us to easily calculate mean AU intensity per condition using .extract_mean():

# Assign each processed frame to the condition it came from
conditions = dict(zip(clip_attrs['input'], clip_attrs['condition']))
fex.sessions = fex['input'].map(conditions)

# Now extracting the mean is grouped by session (experimental condition)
average_au_intensity_per_condition = fex.extract_mean()
average_au_intensity_per_condition.head()

/Users/Esh/Documents/pypackages/py-feat/feat/data.py:1098: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.
  feats.append(pd.Series(v.mean(), name=k))

	mean_frame	mean_FaceRectX	mean_FaceRectY	mean_FaceRectWidth	mean_FaceRectHeight	mean_FaceScore	mean_x_0	mean_x_1	mean_x_2	mean_x_3	...	mean_Pitch	mean_Roll	mean_Yaw	mean_anger	mean_disgust	mean_fear	mean_happiness	mean_sadness	mean_surprise	mean_neutral
badNews	28.089720	324.400708	301.247496	391.827660	532.716661	0.999816	322.016527	328.383112	339.664185	355.110626	...	-5.782593	-3.816183	-2.723817	0.020171	0.014992	0.097006	0.013134	0.086385	0.697528	0.070784
goodNews	20.646766	340.821428	275.474329	387.800142	533.403960	0.999813	341.648032	344.112135	350.285509	359.913069	...	0.752362	-1.497069	-0.174644	0.000287	0.000095	0.004073	0.618644	0.001715	0.371462	0.003724

2 rows × 172 columns

5.4 Evaluting overall AU intensity with a one-sample t-test¶

One analyses we might be interested in is if the average activation of any AUs are significantly higher than .5 (chance) when participants received good news. We can do this by running a one-sample t-test over the mean AU activity per video using the .ttest_1samp() method.

The results suggests more than half the AUs are significantly active beyond chance:

# Set the session of the summary Fe instance to the experiental conditions
average_au_intensity_per_video.sessions = average_au_intensity_per_video.index.map(
    conditions
)

# Then we can perform a test for one condition only
t, p = (
    average_au_intensity_per_video[average_au_intensity_per_video.sessions == 'goodNews']
    .aus()
    .ttest_1samp(0.5)
)

# Show significance start based on bonferroni correction
results = pd.DataFrame(
    {"t": t, "p": p}, index=average_au_intensity_per_video.au_columns
).assign(sig=lambda df: df.p < 0.05 / df.shape[0]).sort_values(by='sig')
results

	t	p	sig
mean_AU01	-3.083360	1.306884e-02	False
mean_AU25	3.540150	6.312618e-03	False
mean_AU24	-2.462283	3.602344e-02	False
mean_AU06	3.189709	1.101172e-02	False
mean_AU23	-0.640517	5.378009e-01	False
mean_AU17	0.021491	9.833231e-01	False
mean_AU28	-2.146230	6.040604e-02	False
mean_AU26	-6.413477	1.233055e-04	True
mean_AU20	-19.486802	1.141779e-08	True
mean_AU15	-8.882221	9.508308e-06	True
mean_AU12	10.020338	3.518224e-06	True
mean_AU11	-15.701840	7.578798e-08	True
mean_AU10	8.198208	1.819405e-05	True
mean_AU09	-12.475848	5.524955e-07	True
mean_AU07	4.371391	1.793454e-03	True
mean_AU05	-8.110445	1.983488e-05	True
mean_AU04	-47.979968	3.718765e-12	True
mean_AU02	-4.691807	1.133177e-03	True
mean_AU14	19.664849	1.053837e-08	True
mean_AU43	-44.777964	6.907718e-12	True

5.5 Comparing conditions¶

5.5.1 Looking at condition ifferences for a single AU using a t-test¶

More likely we want to compare experimental conditions using a two-sample t-test. This allows us to answer separately for each AU: was intensity significantly different when participants received good vs bad news? We can do this using the .ttest_ind() method which takes a column and sessions to compare:

columns2compare = "mean_AU12"
sessions = ("goodNews", "badNews")

t, p = average_au_intensity_per_video.ttest_ind(col=columns2compare, sessions=sessions)
print(f"T-test between {sessions[0]} vs {sessions[1]}: t={t:.2g}, p={p:.3g}")
sns.barplot(
    x=average_au_intensity_per_video.sessions,
    y=columns2compare,
    data=average_au_intensity_per_video,
)

T-test between goodNews vs badNews: t=15, p=9.63e-12

<AxesSubplot:ylabel='mean_AU12'>

5.5.2 Comparing all AUs across conditions using regression¶

More likely we’ll want to run this comparison for all AUs. Because t-tests are just regression we can using the .regress() method along with numerical contrasts to corresponding to the condition means to run a test at every AU.

This is analogous to the “mass-univariate” GLM approach in fMRI researchnd allows us to identify what AUs are significantly more active in condition vs another:

# This time replace the labels for each condition with a numerical contrast code
fex.sessions = fex.input().map(conditions).replace({"goodNews": 0.5, "badNews": -0.5})

# We'll use these contrasts are predictors in our regression
X = pd.DataFrame(fex.sessions)
# Add an intercept to the regression
X["intercept"] = 1

# The dependent variable is AU intensity
y = fex.aus()
# Now we get a t-test at every AU
b, t, p, df, residuals = fex.regress(X=X, y=y)

print("Betas comparing the contrast of good news AU intensity > bad news AU intensity")
results = pd.concat(
    [
        b.round(3).loc[[0]].rename(index={0: "betas"}),
        t.round(3).loc[[0]].rename(index={0: "t-stats"}),
        p.round(3).loc[[0]].rename(index={0: "p-values"}),
    ]
)

results

Betas comparing the contrast of good news AU intensity > bad news AU intensity

/Users/Esh/anaconda3/envs/py-feat/lib/python3.8/site-packages/nltools/stats.py:1076: FutureWarning: Support for multi-dimensional indexing (e.g. `obj[:, None]`) is deprecated and will be removed in a future version.  Convert to a numpy array before indexing instead.
  * sigma[np.newaxis, :]

	AU01	AU02	AU04	AU05	AU06	AU07	AU09	AU10	AU11	AU12	AU14	AU15	AU17	AU20	AU23	AU24	AU25	AU26	AU28	AU43
betas	0.025	0.035	-0.042	-0.168	0.396	0.231	0.123	0.318	0.012	0.469	0.029	0.083	0.064	0.050	-0.033	-0.249	0.247	0.085	-0.008	0.031
t-stats	3.859	6.291	-11.573	-26.156	41.112	30.929	22.875	32.094	8.169	42.164	6.404	14.822	13.344	12.006	-7.492	-21.745	18.964	13.772	-1.538	6.681
p-values	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.124	0.000

5.6 Compare conditions with a classifier and emotion features¶

Another way we can look at differences between conditions is by testing what combination of features can classify between good news or bad news conditions. To do this, we’ll train a Logistc Regression classifier using detected emotion intensity as features. We can use the .predict() method to help us out.

The results suggest that happy expressions have a very high feature weight when predicting good news:

from sklearn.linear_model import LogisticRegression

# Features = emotion predictions for each frame
# Labels = experimental condition the frame belonged to
clf = fex.predict(
    X=fex.emotions(), y=fex.sessions, model=LogisticRegression, solver="liblinear"
)
print(f"score: {clf.score(fex.emotions(), fex.sessions):.3g}")
print(f"coefficients for predicting class: {clf.classes_[1]}")
pd.DataFrame(clf.coef_, columns=fex.emotions().columns)

score: 0.939
coefficients for predicting class: goodNews

	anger	disgust	fear	happiness	sadness	surprise	neutral
0	-0.768841	-0.555749	-2.98863	8.392172	-2.308798	-0.094914	-3.108111

5.7 Time-series correlations¶

Finally we might be interested in looking the similarity of the detected AU time-series across videos, sessions, or people. To compare the similarity of signals over time, we can use the .isc() method which takes an input column. Below we just look at AU 1 and we can get a sense of similar videos were to each other based how AU 1 changed over time.

Warmer colors indicate a pair of videos elicited more similar AU 1 activity over time. We can see just by looking at this AU, how two video cluster seem to emerge reflecting the two experimental conditions: good news and bad news.

# Set sessions back to unique videos
fex.sessions = fex['input']

# ISC returns a video x video pearson correlation matrix
isc = fex.isc(col = "AU01", method='pearson')
sns.heatmap(isc, center=0, vmin=-1, vmax=1, cmap="RdBu_r");

Py-Feat

5. Running a full analysis

Contents