3. Visualizing Facial Expressions¶
In this tutorial we'll explore plotting in Py-Feat using functions from the feat.plotting module along with plotting methods using the Fex data class.
Py-Feat's Detectorv2 produces a dense 478-vertex 3D MediaPipe FaceMesh, and the plotting module can render that mesh directly — in matplotlib 3D or in an interactive Plotly viewport — driven either by AU intensities or by a precomputed mesh. We'll lead with those Detectorv2-style mesh visualizations, then cover the 2D AU→landmark plots, muscle heatmaps, gaze, and animation helpers. A short Legacy: Detectorv1 plots section at the end covers visualizations that depend specifically on the modular Detectorv1 (the xgb AU model and the 68-pt dlib landmark path).
To help visualize facial expressions in a standardized way, Py-Feat includes pre-trained partial-least-squares (PLS) models that map between an array of AU intensities and facial geometry — either the full 478-vertex mesh or the classic 68-pt landmark set.
3.1 Visualizing the full 3D MediaPipe FaceMesh from AU intensities¶
Detectorv2 predicts a dense 478-vertex MediaPipe FaceMesh, and py-feat ships
a matching visualization of that geometry. Pass AU intensities to
plot_face_mesh() and the model predicts a face-shaped mesh in a pose-canonical
frame, then renders it as a 3D wireframe in matplotlib's 3D backend.
By default plot_face_mesh() draws the lighter canonical contours (lips, eyes,
eyebrows, face oval — ~124 edges). Pass mode='tesselation' to draw the full
MediaPipe tessellation (~2,556 edges), which reveals the nose, cheek, and
internal-face structure and makes subtle AU activations easier to see. It
matches the default in the interactive Plotly backend used in §3.3.
This relies on the au_to_mesh PLS model on HuggingFace
(py-feat/au_to_mesh) — downloaded
on first use, then cached.
import numpy as np
import matplotlib.pyplot as plt
from feat.plotting import plot_face_mesh, load_face_mesh_viz_model
mesh_model = load_face_mesh_viz_model()
au_columns = mesh_model.au_columns
print('AU columns the model expects:', au_columns)
rest = np.zeros(20, dtype=np.float32)
smile = np.zeros(20, dtype=np.float32)
smile[au_columns.index('AU12')] = 3.0
brow = np.zeros(20, dtype=np.float32)
brow[au_columns.index('AU04')] = 3.0
_fig = plt.figure(figsize=(15, 5))
for _i, (label, au) in enumerate([('Rest', rest), ('AU12 smile', smile), ('AU04 brow lower', brow)]):
_ax = _fig.add_subplot(1, 3, _i + 1, projection='3d')
plot_face_mesh(au=au, ax=_ax, model=mesh_model, mode='tesselation')
_ax.set_title(label)
plt.tight_layout()
_fig
AU columns the model expects: ['AU01', 'AU02', 'AU04', 'AU05', 'AU06', 'AU07', 'AU09', 'AU10', 'AU11', 'AU12', 'AU14', 'AU15', 'AU17', 'AU20', 'AU23', 'AU24', 'AU25', 'AU26', 'AU28', 'AU43']
3.2 Interactive 3D visualization with Plotly¶
For an interactive 3D viewport you can rotate, pan, and zoom (especially
useful in Jupyter notebooks), plot_face_mesh_plotly() returns a
plotly.graph_objects.Figure instead of a matplotlib axis. The mode='tesselation'
default draws the full 2,556-edge MP tessellation for a dense 3D look; pass
mode='contours' for the same canonical-features wireframe as the matplotlib
version.
The function takes the same au= / mesh= source dispatch as plot_face_mesh,
so you can use it with either AU intensities or a precomputed mesh.
from feat.plotting import plot_face_mesh_plotly
# Same smile activation as 3.1
fig_plotly = plot_face_mesh_plotly(au=smile, mode="tesselation")
fig_plotly.update_layout(width=500, height=500)
fig_plotly
You can also persist the figure to standalone HTML for sharing or to
PNG via Plotly's write_image() method (which uses kaleido under the hood):
For the lighter, contours-only view that matches plot_face_mesh:
Adding a 3D gaze arrow¶
Both plot_face_mesh and plot_face_mesh_plotly accept a gaze=(pitch, yaw)
tuple of head-centric angles (radians), matching the format the gaze model outputs in
fex.gaze_pitch / fex.gaze_yaw. A yellow arrow is drawn from the
outer-canthi midpoint in the mesh's pose-canonical frame, scaled to
gaze_length_frac (default 30%) of face height.
Forward gaze (pitch=0, yaw=0) points along +Z (out of the face), so in the default front-on plotly camera it appears as a small point — drag to rotate the camera if you want to see it as an arrow.
fig_gaze = plot_face_mesh_plotly(au=None, mode='tesselation', gaze=(np.deg2rad(15), np.deg2rad(20)))
fig_gaze.update_layout(width=500, height=500, title_text='Mesh + 3D gaze arrow')
# Pitch ~+15° (looking up), yaw ~+20° (eyes drift toward viewer's right).
# Pass radians; the gaze detector output is already in radians so you can
# plug fex.gaze_pitch / fex.gaze_yaw straight in. Tesselation mode shows
# enough of the face (nose, cheeks, eyes) for the gaze arrow to read
# anatomically — contours leaves too few landmarks for the eye to be
# obvious.
fig_gaze
3.3 Animating the 3D MediaPipe FaceMesh¶
animate_face_mesh_plotly() returns a Plotly Figure with
play/pause/loop buttons and a per-frame slider, and the camera stays
rotatable while the animation is playing. So you can rotate to a
profile view, hit play, and watch the expression morph from that vantage.
It uses the same interpolate_aus cubic-easing helper as the 2D
animate_face, and the same mode='tesselation' / 'contours' knob as
plot_face_mesh_plotly. By default it appends a reverse pass so the
animation returns to the starting expression on each cycle.
Below we animate a smile starting from rest in tesselation mode — the
dense edge set shows the nose, cheek, and inner-face structure that
makes the face recognizable. Pass mode='contours' if you need a
smaller embedded HTML (~500 KB vs several MB for tesselation).
from feat.plotting import animate_face_mesh_plotly
# Animate from rest → smile → rest. Tesselation mode (default) is much
# more recognizable than contours — contours shows only a lips/eyelids/
# brows/face oval, missing nose, cheeks, iris, etc. Tradeoff is HTML
# output size (~5-10 MB for 24 frames vs ~500 KB for contours).
fig_anim = animate_face_mesh_plotly(
start=rest,
end=smile,
num_frames=12,
fps=15,
mode="tesselation",
)
fig_anim.update_layout(width=500, height=600, title_text="AU12 smile (rest → peak → rest)")
fig_anim
3.4 The AU atlas: all 20 AUs as mesh panels¶
A quick way to build intuition for what each Action Unit does to the face is to
drive the au_to_mesh model with one AU at a time and render the whole atlas as
a grid. Below we activate each of the 20 AUs to intensity 1.5 and draw the
resulting mesh (colored by per-vertex displacement from neutral, plasma
colormap) on top of the faint neutral mesh (gray). Brighter regions move more.
The mesh comes out in a pose-canonical 3D frame; here we take a simple front projection (x lateral, y vertical) and flip the vertical axis when needed so the forehead sits above the chin.
from matplotlib.collections import LineCollection
from feat.plotting import predict_face_mesh
from feat.utils.mp_plotting import FaceLandmarksConnections
mesh_au_cols = list(mesh_model.au_columns)
mesh_edges = np.array(
[[c.start, c.end] for c in FaceLandmarksConnections.FACE_LANDMARKS_TESSELATION]
)
def project_mesh_2d(mesh):
# Front view; flip vertical so forehead (vertex 10) is above chin (152).
xy = mesh[:, :2].copy()
if xy[10, 1] < xy[152, 1]:
xy[:, 1] = -xy[:, 1]
return xy
mesh_neutral2d = project_mesh_2d(
predict_face_mesh(np.zeros(len(mesh_au_cols), np.float32), mesh_model)
)
_n = len(mesh_au_cols)
_ncol = 5
_nrow = (_n + _ncol - 1) // _ncol
_fig, _axes = plt.subplots(_nrow, _ncol, figsize=(3 * _ncol, 3 * _nrow))
for _i, _au in enumerate(mesh_au_cols):
_v = np.zeros(_n, np.float32)
_v[_i] = 1.5
_p2 = project_mesh_2d(predict_face_mesh(_v, mesh_model))
_mag = np.linalg.norm(_p2 - mesh_neutral2d, axis=1)
_ax = _axes.flat[_i]
_ax.add_collection(
LineCollection(mesh_neutral2d[mesh_edges], colors="lightgray", lw=0.25, alpha=0.5)
)
_ax.add_collection(
LineCollection(
_p2[mesh_edges], array=_mag[mesh_edges].mean(1), cmap="plasma", lw=0.5, alpha=0.9
)
)
_ax.autoscale()
_ax.set_aspect("equal")
_ax.axis("off")
_ax.set_title(_au, fontsize=9)
for _j in range(_n, _nrow * _ncol):
_axes.flat[_j].axis("off")
_fig.suptitle("au_to_mesh — each AU @1.5 (neutral gray, activated colored by displacement)")
_fig.tight_layout()
_fig