1. Detecting facial expressions from images¶
In this tutorial we'll use Detectorv2 — Py-Feat's single multi-task model — to detect faces, landmarks, action units, emotions, valence/arousal, gaze, and more from images, and to visualize the results. At the end we'll cover the modular Detectorv1 for when you want to swap or disable individual models.
1.1 Setting up a detector¶
The recommended way to extract facial features in Py-Feat 0.7+ is Detectorv2 — a single multi-task neural network that, in one forward pass, predicts Action Units, emotions, valence/arousal, gaze, head pose, 68-point landmarks, and a 478-point 3D MediaPipe FaceMesh. It's fast (especially on single frames) and is what the rest of this tutorial uses. Passing identity_model="arcface" also adds a face-identity embedding.
The first time you initialize a detector, Py-Feat downloads the required pretrained weights from our HuggingFace Repository and caches them to disk; subsequent runs reuse the cached weights.
You can find a list of default models on this page. For the older modular detector, see the Using the modular Detectorv1 section at the end of this tutorial.
from feat import Detectorv2
# One multi-task model: AUs, emotions, valence/arousal, gaze, head pose,
# 68-pt landmarks, and a 478-pt 3D FaceMesh. identity_model="arcface" adds a
# face-identity embedding. device was selected above (cuda/mps/cpu).
detector_v2 = Detectorv2(device=device, identity_model="arcface")
1.2 Processing a single image¶
Let's process a single image with a single face. Py-feat includes a demo image for this purpose called single_face.jpg so lets use that. You can also use the convenient imshow function which will automatically load an image into a numpy array if provided a path unlike matplotlib:
from feat.utils.io import get_test_data_path
from feat.plotting import imshow
import os
# Helper to point to the test data folder
test_data_dir = get_test_data_path()
# Get the full path
single_face_img_path = os.path.join(test_data_dir, "single_face.jpg")
# Plot it
imshow(single_face_img_path)
Now we use our initialized detector instance to make predictions with the .detect() method, passing data_type="image". This is the main workhorse method that will perform face, landmark, au, and emotion detection using the loaded models. It always returns a Fex data instance:
single_face_prediction = detector_v2.detect(single_face_img_path, data_type="image")
type(single_face_prediction) # instance of a Fex class
# Show results
single_face_prediction
0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.68s/it] 100%|██████████| 1/1 [00:02<00:00, 2.69s/it]
| FaceRectX | FaceRectY | FaceRectWidth | FaceRectHeight | FaceScore | x_0 | x_1 | x_2 | x_3 | x_4 | ... | mouthStretchRight | mouthUpperUpLeft | mouthUpperUpRight | noseSneerLeft | noseSneerRight | FrameHeight | FrameWidth | input | frame | Identity | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 129.789062 | 118.849884 | 301.164368 | 301.164368 | 0.999925 | 191.551285 | 193.610031 | 195.374664 | 198.903931 | 204.197845 | ... | 0.016357 | 0.710938 | 0.632812 | 0.000002 | 0.000003 | 562.0 | 572.0 | /home/ljchang/Github/py-feat/feat/tests/data/single_face.jpg | 0 | Person_0 |
1 rows × 2182 columns
1.3 Working with Fex outputs¶
The output of any detection always returns a Fex data class instance. This class is a lightweight wrapper around a pandas dataframe that contains columns with values for detection type.
So you can use any pandas methods you're already familiar with:
# We always return a dataframe even if there's just a single row,
# i.e. no Series
single_face_prediction.head()
| FaceRectX | FaceRectY | FaceRectWidth | FaceRectHeight | FaceScore | x_0 | x_1 | x_2 | x_3 | x_4 | ... | mouthStretchRight | mouthUpperUpLeft | mouthUpperUpRight | noseSneerLeft | noseSneerRight | FrameHeight | FrameWidth | input | frame | Identity | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 129.789062 | 118.849884 | 301.164368 | 301.164368 | 0.999925 | 191.551285 | 193.610031 | 195.374664 | 198.903931 | 204.197845 | ... | 0.016357 | 0.710938 | 0.632812 | 0.000002 | 0.000003 | 562.0 | 572.0 | /home/ljchang/Github/py-feat/feat/tests/data/single_face.jpg | 0 | Person_0 |
1 rows × 2182 columns
Fex provides convenient attributes to access specific groups of columns so you don't have to write a bunch of pandas code to get the data you need:
| FaceRectX | FaceRectY | FaceRectWidth | FaceRectHeight | FaceScore | |
|---|---|---|---|---|---|
| 0 | 129.789062 | 118.849884 | 301.164368 | 301.164368 | 0.999925 |
| AU01 | AU02 | AU04 | AU05 | AU06 | AU07 | AU09 | AU10 | AU11 | AU12 | AU14 | AU15 | AU17 | AU20 | AU23 | AU24 | AU25 | AU26 | AU28 | AU43 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.044996 | 0.122796 | 0.001304 | 0.0 | 0.82804 | 0.316168 | 0.009288 | 0.977741 | 0.918384 | 0.991344 | 0.013961 | 0.090856 | 0.0 | 0.0 | 0.099998 | 0.021641 | 0.994322 | 0.34807 | 0.130655 | 0.010609 |
| Neutral | Happy | Sad | Surprise | Fear | Disgust | Anger | |
|---|---|---|---|---|---|---|---|
| 0 | 0.010986 | 0.980469 | 0.000595 | 0.003159 | 0.000151 | 0.002762 | 0.000315 |
Detectorv2 also predicts continuous valence (unpleasant → pleasant) and
arousal (calm → excited) — the two affective dimensions the modular v1
Detectorv1 does not produce. They're plain Fex columns:
| valence | arousal | |
|---|---|---|
| 0 | 0.730469 | 0.081543 |
| Pitch | Roll | Yaw | X | Y | Z | |
|---|---|---|---|---|---|---|
| 0 | 0.014343 | 0.038086 | 0.177734 | 0.010132 | 0.228516 | 5.15625 |
| Identity | |
|---|---|
| 0 | Person_0 |
1.4 Saving and Loading detections from a file¶
Since a Fex object is just a sub-classed DataFrames we can use the .to_csv method to save our detections toa file:
To create a new Fex instance from a csv file use our custom read_feat() function instead pf pd.read_csv:
from feat.utils.io import read_feat
input_prediction = read_feat("output.csv")
# We can quick access features like before
input_prediction.aus
| AU01 | AU02 | AU04 | AU05 | AU06 | AU07 | AU09 | AU10 | AU11 | AU12 | AU14 | AU15 | AU17 | AU20 | AU23 | AU24 | AU25 | AU26 | AU28 | AU43 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.044996 | 0.122796 | 0.001304 | 0.0 | 0.82804 | 0.316168 | 0.009288 | 0.977741 | 0.918384 | 0.991344 | 0.013961 | 0.090856 | 0.0 | 0.0 | 0.099998 | 0.021641 | 0.994322 | 0.34807 | 0.130655 | 0.010609 |
Real-time saving during detection (low-memory mode)¶
You can also write Fex outputs to a file during detection by passing a save argument to detect. This will save the Fex output to a csv file every time a face is detected.
This can be useful when processing multiple images or videos (as we'll see later).
fex = detector_v2.detect(inputs=single_face_img_path, data_type="image", save='detections.csv')
fex.head()
0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 37.02it/s]
| FaceRectX | FaceRectY | FaceRectWidth | FaceRectHeight | FaceScore | x_0 | x_1 | x_2 | x_3 | x_4 | ... | mouthStretchRight | mouthUpperUpLeft | mouthUpperUpRight | noseSneerLeft | noseSneerRight | FrameHeight | FrameWidth | input | frame | Identity | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 129.789062 | 118.849884 | 301.164368 | 301.164368 | 0.999925 | 191.551285 | 193.610031 | 195.374664 | 198.903931 | 204.197845 | ... | 0.016357 | 0.710938 | 0.632812 | 0.000002 | 0.000003 | 562.0 | 572.0 | /home/ljchang/Github/py-feat/feat/tests/data/single_face.jpg | 0 | Person_0 |
1 rows × 2182 columns
We can use our terminal to see that detections.csv exists and contains the same content as fex
FaceRectX,FaceRectY,FaceRectWidth,FaceRectHeight,FaceScore,x_0,x_1,x_2,x_3,x_4,x_5,x_6,x_7,x_8,x_9,x_10,x_11,x_12,x_13,x_14,x_15,x_16,x_17,x_18,x_19,x_20,x_21,x_22,x_23,x_24,x_25,x_26,x_27,x_28,x_29,x_30,x_31,x_32,x_33,x_34,x_35,x_36,x_37,x_38,x_39,x_40,x_41,x_42,x_43,x_44,x_45,x_46,x_47,x_48,x_49,x_50,x_51,x_52,x_53,x_54,x_55,x_56,x_57,x_58,x_59,x_60,x_61,x_62,x_63,x_64,x_65,x_66,x_67,y_0,y_1,y_2,y_3,y_4,y_5,y_6,y_7,y_8,y_9,y_10,y_11,y_12,y_13,y_14,y_15,y_16,y_17,y_18,y_19,y_20,y_21,y_22,y_23,y_24,y_25,y_26,y_27,y_28,y_29,y_30,y_31,y_32,y_33,y_34,y_35,y_36,y_37,y_38,y_39,y_40,y_41,y_42,y_43,y_44,y_45,y_46,y_47,y_48,y_49,y_50,y_51,y_52,y_53,y_54,y_55,y_56,y_57,y_58,y_59,y_60,y_61,y_62,y_63,y_64,y_65,y_66,y_67,Pitch,Roll,Yaw,X,Y,Z,AU01,AU02,AU04,AU05,AU06,AU07,AU09,AU10,AU11,AU12,AU14,AU15,AU17,AU20,AU23,AU24,AU25,AU26,AU28,AU43,anger,disgust,fear,happiness,sadness,surprise,neutral,Identity_1,Identity_2,Identity_3,Identity_4,Identity_5,Identity_6,Identity_7,Identity_8,Identity_9,Identity_10,Identity_11,Identity_12,Identity_13,Identity_14,Identity_15,Identity_16,Identity_17,Identity_18,Identity_19,Identity_20,Identity_21,Identity_22,Identity_23,Identity_24,Identity_25,Identity_26,Identity_27,Identity_28,Identity_29,Identity_30,Identity_31,Identity_32,Identity_33,Identity_34,Identity_35,Identity_36,Identity_37,Identity_38,Identity_39,Identity_40,Identity_41,Identity_42,Identity_43,Identity_44,Identity_45,Identity_46,Identity_47,Identity_48,Identity_49,Identity_50,Identity_51,Identity_52,Identity_53,Identity_54,Identity_55,Identity_56,Identity_57,Identity_58,Identity_59,Identity_60,Identity_61,Identity_62,Identity_63,Identity_64,Identity_65,Identity_66,Identity_67,Identity_68,Identity_69,Identity_70,Identity_71,Identity_72,Identity_73,Identity_74,Identity_75,Identity_76,Identity_77,Identity_78,Identity_79,Identity_80,Identity_81,Identity_82,Identity_83,Identity_84,Identity_85,Identity_86,Identity_87,Identity_88,Identity_89,Identity_90,Identity_91,Identity_92,Identity_93,Identity_94,Identity_95,Identity_96,Identity_97,Identity_98,Identity_99,Identity_100,Identity_101,Identity_102,Identity_103,Identity_104,Identity_105,Identity_106,Identity_107,Identity_108,Identity_109,Identity_110,Identity_111,Identity_112,Identity_113,Identity_114,Identity_115,Identity_116,Identity_117,Identity_118,Identity_119,Identity_120,Identity_121,Identity_122,Identity_123,Identity_124,Identity_125,Identity_126,Identity_127,Identity_128,Identity_129,Identity_130,Identity_131,Identity_132,Identity_133,Identity_134,Identity_135,Identity_136,Identity_137,Identity_138,Identity_139,Identity_140,Identity_141,Identity_142,Identity_143,Identity_144,Identity_145,Identity_146,Identity_147,Identity_148,Identity_149,Identity_150,Identity_151,Identity_152,Identity_153,Identity_154,Identity_155,Identity_156,Identity_157,Identity_158,Identity_159,Identity_160,Identity_161,Identity_162,Identity_163,Identity_164,Identity_165,Identity_166,Identity_167,Identity_168,Identity_169,Identity_170,Identity_171,Identity_172,Identity_173,Identity_174,Identity_175,Identity_176,Identity_177,Identity_178,Identity_179,Identity_180,Identity_181,Identity_182,Identity_183,Identity_184,Identity_185,Identity_186,Identity_187,Identity_188,Identity_189,Identity_190,Identity_191,Identity_192,Identity_193,Identity_194,Identity_195,Identity_196,Identity_197,Identity_198,Identity_199,Identity_200,Identity_201,Identity_202,Identity_203,Identity_204,Identity_205,Identity_206,Identity_207,Identity_208,Identity_209,Identity_210,Identity_211,Identity_212,Identity_213,Identity_214,Identity_215,Identity_216,Identity_217,Identity_218,Identity_219,Identity_220,Identity_221,Identity_222,Identity_223,Identity_224,Identity_225,Identity_226,Identity_227,Identity_228,Identity_229,Identity_230,Identity_231,Identity_232,Identity_233,Identity_234,Identity_235,Identity_236,Identity_237,Identity_238,Identity_239,Identity_240,Identity_241,Identity_242,Identity_243,Identity_244,Identity_245,Identity_246,Identity_247,Identity_248,Identity_249,Identity_250,Identity_251,Identity_252,Identity_253,Identity_254,Identity_255,Identity_256,Identity_257,Identity_258,Identity_259,Identity_260,Identity_261,Identity_262,Identity_263,Identity_264,Identity_265,Identity_266,Identity_267,Identity_268,Identity_269,Identity_270,Identity_271,Identity_272,Identity_273,Identity_274,Identity_275,Identity_276,Identity_277,Identity_278,Identity_279,Identity_280,Identity_281,Identity_282,Identity_283,Identity_284,Identity_285,Identity_286,Identity_287,Identity_288,Identity_289,Identity_290,Identity_291,Identity_292,Identity_293,Identity_294,Identity_295,Identity_296,Identity_297,Identity_298,Identity_299,Identity_300,Identity_301,Identity_302,Identity_303,Identity_304,Identity_305,Identity_306,Identity_307,Identity_308,Identity_309,Identity_310,Identity_311,Identity_312,Identity_313,Identity_314,Identity_315,Identity_316,Identity_317,Identity_318,Identity_319,Identity_320,Identity_321,Identity_322,Identity_323,Identity_324,Identity_325,Identity_326,Identity_327,Identity_328,Identity_329,Identity_330,Identity_331,Identity_332,Identity_333,Identity_334,Identity_335,Identity_336,Identity_337,Identity_338,Identity_339,Identity_340,Identity_341,Identity_342,Identity_343,Identity_344,Identity_345,Identity_346,Identity_347,Identity_348,Identity_349,Identity_350,Identity_351,Identity_352,Identity_353,Identity_354,Identity_355,Identity_356,Identity_357,Identity_358,Identity_359,Identity_360,Identity_361,Identity_362,Identity_363,Identity_364,Identity_365,Identity_366,Identity_367,Identity_368,Identity_369,Identity_370,Identity_371,Identity_372,Identity_373,Identity_374,Identity_375,Identity_376,Identity_377,Identity_378,Identity_379,Identity_380,Identity_381,Identity_382,Identity_383,Identity_384,Identity_385,Identity_386,Identity_387,Identity_388,Identity_389,Identity_390,Identity_391,Identity_392,Identity_393,Identity_394,Identity_395,Identity_396,Identity_397,Identity_398,Identity_399,Identity_400,Identity_401,Identity_402,Identity_403,Identity_404,Identity_405,Identity_406,Identity_407,Identity_408,Identity_409,Identity_410,Identity_411,Identity_412,Identity_413,Identity_414,Identity_415,Identity_416,Identity_417,Identity_418,Identity_419,Identity_420,Identity_421,Identity_422,Identity_423,Identity_424,Identity_425,Identity_426,Identity_427,Identity_428,Identity_429,Identity_430,Identity_431,Identity_432,Identity_433,Identity_434,Identity_435,Identity_436,Identity_437,Identity_438,Identity_439,Identity_440,Identity_441,Identity_442,Identity_443,Identity_444,Identity_445,Identity_446,Identity_447,Identity_448,Identity_449,Identity_450,Identity_451,Identity_452,Identity_453,Identity_454,Identity_455,Identity_456,Identity_457,Identity_458,Identity_459,Identity_460,Identity_461,Identity_462,Identity_463,Identity_464,Identity_465,Identity_466,Identity_467,Identity_468,Identity_469,Identity_470,Identity_471,Identity_472,Identity_473,Identity_474,Identity_475,Identity_476,Identity_477,Identity_478,Identity_479,Identity_480,Identity_481,Identity_482,Identity_483,Identity_484,Identity_485,Identity_486,Identity_487,Identity_488,Identity_489,Identity_490,Identity_491,Identity_492,Identity_493,Identity_494,Identity_495,Identity_496,Identity_497,Identity_498,Identity_499,Identity_500,Identity_501,Identity_502,Identity_503,Identity_504,Identity_505,Identity_506,Identity_507,Identity_508,Identity_509,Identity_510,Identity_511,Identity_512,gaze_pitch,gaze_yaw,gaze_angle,FrameHeight,FrameWidth,input,frame,Identity 173.0,118.0,214.0,302.0,0.99992526,189.7375,192.03098,194.9403,199.4053,209.22238,224.58171,242.54472,260.50235,280.2223,301.36307,322.05557,341.08475,356.54276,365.74463,369.91147,372.7406,374.88477,200.97452,211.08772,224.63707,239.18036,253.3822,285.34732,301.3941,318.26166,334.3872,347.92035,269.0178,268.667,268.16354,267.7119,250.47733,259.65778,269.79153,281.16534,291.77374,218.09608,229.27466,238.92705,247.95518,238.60841,228.8382,295.97256,305.9265,316.20093,328.16815,316.6493,306.42853,237.73685,249.9643,262.80682,272.76385,283.53204,300.95755,319.13538,303.37262,287.01843,274.6242,263.63232,250.37994,241.68594,263.10797,273.26324,284.54544,315.14075,285.55054,273.74115,263.06696,243.90921,272.37964,300.358,328.15558,352.09137,369.96106,383.4886,395.3938,398.9548,395.36267,383.9172,369.41486,348.9672,322.38892,292.34998,262.12326,231.08711,232.9226,220.77203,216.82547,217.83142,222.4873,219.8429,214.50748,212.93475,217.29068,227.9,243.6503,262.00995,279.3964,296.68823,305.68634,309.9701,313.10114,308.9636,304.1381,249.30417,245.2457,244.91533,249.75975,251.16226,251.164,248.37338,242.27219,242.03772,244.73022,247.52689,248.74739,329.81097,322.73257,319.89734,321.69507,319.14752,320.42932,324.18005,342.61877,351.19342,352.87253,351.55414,343.98422,330.47723,328.07892,329.11435,327.06836,325.34338,338.65643,340.04712,339.11865,0.09432173,0.10586592,-0.028590795,-0.018647293,0.1973652,6.553823,0.44878685,0.4206227,0.27863425,0.38800073,0.84491813,1.0,0.5580089,0.023500565,1.0,0.8620931,0.64808524,0.23444104,0.45146358,0.0,0.09711652,0.4737867,0.77049214,0.24348351,0.069186345,0.7194142,0.00014707568,1.4417356e-05,2.949324e-06,0.99725515,0.0024483204,1.9043113e-05,0.00011313761,0.84382164,-2.1944287,1.4424969,0.8901416,-0.49464354,-1.5290192,1.0275708,0.4551492,-0.062250737,0.6320777,-0.9637744,0.57443464,0.026079873,1.5730002,-0.47371083,-1.3710012,-1.5472629,-0.31947416,-0.29019263,0.5881625,1.4226279,0.9124276,1.6321814,-1.235631,0.1595744,0.7689769,-1.5274576,0.27129808,-0.82948357,-0.25093922,1.1154202,-0.031148659,-0.10630518,0.20665774,0.8048497,0.3913627,0.5033297,0.19042148,1.6690123,-0.5240457,-0.08604671,-0.49802718,2.0181186,0.41690108,-0.4909794,-0.17116752,2.0086563,-0.033448137,-1.1527746,-0.7043438,0.27357537,0.024206752,-0.5541352,0.61359125,1.102979,1.0176136,-1.5130823,1.5636501,1.9901445,-0.3455308,0.2508021,-1.8781825,-0.285034,-0.09256218,-0.30566323,-2.1582968,-0.9177012,0.8052693,0.04293533,-0.5363763,-1.515096,0.15786588,-0.5689566,0.32093555,-0.4313593,-0.744562,-0.36107683,-0.24286595,0.28637028,-0.9653402,-1.4981791,0.4254374,-1.2562336,0.5493574,-0.5439653,-0.2219203,0.117445424,0.16015461,-1.0769112,-1.004527,0.837847,-0.9578061,-1.374805,0.83603454,-0.6067281,-0.21321306,-0.71894574,0.8139959,0.69799274,0.57672185,0.012147394,-0.090089284,-1.2394941,1.6852871,1.810418,0.42154086,0.37449223,-0.70340747,0.8491017,0.23594764,0.29021084,0.035794668,-0.3586311,1.029334,1.4683076,-0.89666706,0.26130068,-1.0890791,-0.9934982,-0.32229796,0.26891816,-1.1245471,-1.4923224,0.651493,-0.1348436,0.48411494,1.6316946,-0.9764149,0.5983776,-1.0359848,-0.77293915,0.0061091986,-0.8197774,-1.3336244,-0.0013608827,0.48552278,-1.7343211,1.0336149,0.26520044,0.12911682,0.63865024,-0.88401526,-1.7226863,-0.31724983,0.8062583,0.40793782,-0.33456522,-0.6055689,0.52793586,0.49572262,1.4866506,0.5555142,1.6883723,0.06264587,2.2374842,0.25643864,0.31495512,-0.60136235,0.9957762,1.4601386,-0.48355666,0.5183803,1.0345165,0.35851192,0.6224214,0.13155851,-0.12119161,-0.068975054,0.69345385,0.8254745,-0.35175425,-0.7194306,0.66448003,-0.08241021,0.6939616,0.19503774,-0.73789144,-1.647881,0.5466219,-0.4330851,-0.65389085,-0.12298704,1.9666555,0.20032936,-2.0186498,-0.44715336,0.7622531,0.100309685,1.8380036,0.99815136,0.22691074,0.80910575,0.79005486,1.252872,-0.4298596,2.255692,-0.5850849,0.3229892,2.1970592,-0.31037018,0.7575922,0.8034304,-0.60109276,-1.0258784,0.37946454,0.83619153,-0.16068223,1.1635728,-0.7721407,-0.04534028,-0.19490203,0.572779,-0.34251156,0.5704242,-0.4050599,1.3312846,-3.0493586,-0.1911788,1.2379364,-1.3413533,-2.4459374,0.69507754,0.30033308,0.44096595,1.5210934,-0.090744175,-0.064905114,-0.8144855,1.3826928,0.74472946,-0.48424482,-1.6877656,1.6587565,0.0269598,1.1989529,0.7142111,0.4660395,-0.75749606,-0.4693725,-0.23894854,0.09611911,0.45768157,-0.06328009,-0.9296423,-0.9734718,1.0688844,0.24748528,0.6086135,0.31095338,-1.636395,-0.042632066,-0.20853157,-1.8361392,-0.10906421,-1.2232193,0.28551796,-0.54056406,1.0269338,-1.93602,-0.7192052,-0.7382168,-0.71725863,-0.0013584814,0.17075773,0.34375298,-1.0362245,-1.1420664,1.565184,0.07979477,-0.73953545,0.94350195,1.3623079,0.9681605,-0.0092139505,-1.9868705,0.41898763,0.027894095,-0.78892535,-0.38581476,-0.33598933,-0.82006973,0.44960135,-1.008856,-0.6672202,0.934044,1.8545943,-0.9332076,-0.25765416,-1.9497443,1.0461007,1.4879789,-0.4341537,-1.2908771,-0.81122184,1.5324521,2.0066519,0.085615955,-0.84062904,0.99551237,0.9235111,1.4485908,0.8604149,0.57535315,-0.69420016,-1.3931078,0.9288821,-0.49175978,-0.47318757,-0.5138363,0.11929259,-0.75242525,-1.4709724,-0.6844049,1.1197972,1.2981364,-2.1171677,0.8747587,-2.4839768,-0.12061673,0.81449944,0.15604722,1.168792,-1.3050436,-2.157722,-0.12875101,-0.1412873,1.0253664,0.1839306,-0.112758964,-0.9635113,-0.43794897,1.606536,0.93744934,-0.5772973,0.57813853,-0.45181355,-1.072451,1.5449775,0.29602522,0.9234261,-0.48433077,0.5212904,0.26000908,0.40246472,-0.49173614,-0.6729331,1.715049,0.5015819,1.152203,1.0454214,0.25881588,-0.88372946,1.4202816,1.6287389,-1.4087504,0.9857841,-1.5985304,-1.1971784,1.7973818,0.45279852,0.6603103,-1.4127369,-0.99995613,-0.29921213,-0.2903856,0.17830883,-0.060993947,0.2655129,-1.8369479,0.07138811,-0.5103997,-1.5700098,-1.2769625,-0.5700408,-0.7611566,0.123295486,0.2943317,0.75698215,0.10180854,0.7316171,-0.35568646,-0.6535605,-1.345554,-0.027899459,1.2022258,-0.23051368,-1.5061866,-0.052969,0.16803616,-0.6869088,-1.1532884,1.6778353,-0.3727866,0.34981456,-0.7535336,1.0477397,-0.27796754,0.46623024,-0.5882883,-1.6574335,0.53881377,0.36865574,-0.044367503,0.44819087,1.2828245,-0.37670058,0.55130035,-2.6826935,0.2227352,0.66616774,1.9966779,1.4605901,-1.5789092,-1.6089616,-1.994954,1.2678417,0.06002782,1.2076802,0.054732982,-0.7285239,-0.7881693,-1.0598147,-0.42157444,0.22311097,-0.116378054,-0.10830915,0.21812901,-1.7285911,1.9326785,0.5936302,-1.5281547,-0.9006205,0.6155793,-2.1618135,0.4265087,-0.048697144,1.0910097,-0.41238406,0.4493067,0.71419644,0.42905426,0.29693654,-1.001604,0.4175642,-1.5052927,0.0035221153,0.4851495,-1.7788712,1.7945775,0.556476,0.22368866,0.75182307,-0.8736081,0.18163975,-0.46980342,0.109837756,-0.2805096,-1.0180806,0.5159203,-0.3640671,0.6631608,-1.200378,1.204272,0.63779706,0.06899229,1.225242,-1.1944352,-0.43380395,1.0515643,-0.8950867,-0.3245322,0.5388654,0.24384916,-1.3286006,-0.047712356,-0.6671343,-0.123022124,-0.41009492,0.86663425,-0.014817471,-1.4173596,0.008055719,-0.4432219,0.0065221186,-0.11003633,0.21240766,-0.32700697,-0.063415304,-0.32913876,1.8566418,-0.45959014,0.07751028,-0.6448785,1.7830682,0.99433017,0.64155316,0.6939455,0.21616611,0.050468393,0.36110032,0.31943572,-1.9522852,0.72673106,0.5510022,-0.6396912,-0.46112406,0.8178305,1.4126265,-0.7291982,-1.5748446,0.2998454,1.8405756,-0.04392211,-0.80184525,0.802778,562.0,572.0,/home/ljchang/Github/py-feat/feat/tests/data/single_face.jpg,0,Person_0
CompletedProcess(args='head detections.csv', returncode=0)
1.5 Visualizing detection results.¶
Fex objects have a method called .plot_detections() to generate a summary figure of detected faces, action units and emotions. It always returns a list of matplotlib figures:
Overlaying gaze direction¶
plot_detections(gazes=True) overlays a yellow arrow on each detected
face showing where it's looking. The arrow direction comes from
gaze_pitch / gaze_yaw columns produced by whichever gaze model is
active — in v0.7+ the default is L2CS (Abdelrahman et al. 2022, a
ResNet50 trained on Gaze360 + MPIIGaze). Angles are in radians,
head-centric: positive pitch = looking up, positive yaw = subject's
gaze drifts toward the viewer's right.
# fex.gaze_columns lists which columns hold the gaze model's output;
# for L2CS that's gaze_pitch and gaze_yaw (radians).
print('gaze columns:', single_face_prediction.gaze_columns)
print(single_face_prediction[['gaze_pitch', 'gaze_yaw']])
_figs = single_face_prediction.plot_detections(faces='landmarks', gazes=True, muscles=False)
_figs[0]
gaze columns: ['gaze_pitch', 'gaze_yaw', 'gaze_angle'] gaze_pitch gaze_yaw 0 0.024902 0.059326
1.6 Detecting multiple faces from a single image¶
A Detectorv1 will automatically find multiple faces in a single image and will create 1 row per detected face in the Fex object it outputs.
Notice how image_prediction is now a Fex instance with 5 rows, one for each detected face. We can confirm this by plotting our detection results like before:
multi_face_image_path = os.path.join(test_data_dir, "multi_face.jpg")
multi_face_prediction = detector_v2.detect(multi_face_image_path, data_type="image")
# Show results
multi_face_prediction
0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:01<00:00, 1.57s/it] 100%|██████████| 1/1 [00:01<00:00, 1.57s/it]
| FaceRectX | FaceRectY | FaceRectWidth | FaceRectHeight | FaceScore | x_0 | x_1 | x_2 | x_3 | x_4 | ... | mouthStretchRight | mouthUpperUpLeft | mouthUpperUpRight | noseSneerLeft | noseSneerRight | FrameHeight | FrameWidth | input | frame | Identity | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 656.876831 | 277.657898 | 158.751953 | 158.751953 | 0.999982 | 684.472412 | 683.852234 | 683.387146 | 683.697205 | 685.402588 | ... | 0.049561 | 0.832031 | 0.843750 | 0.000002 | 4.500151e-06 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 0 | Person_0 |
| 1 | 505.674805 | 297.692688 | 149.074585 | 149.074585 | 0.999973 | 536.101135 | 535.082092 | 534.208618 | 534.063049 | 534.790955 | ... | 0.221680 | 0.792969 | 0.761719 | 0.000004 | 2.905726e-06 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 0 | Person_1 |
| 2 | 288.728638 | 222.224716 | 149.330963 | 149.330948 | 0.999939 | 313.957397 | 314.686554 | 316.436523 | 319.061493 | 322.998932 | ... | 0.034668 | 0.703125 | 0.707031 | 0.000002 | 3.963709e-06 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 0 | Person_2 |
| 3 | 197.528900 | 57.485100 | 130.303192 | 130.303162 | 0.999205 | 217.761520 | 216.361786 | 215.471039 | 215.089294 | 215.852783 | ... | 0.114258 | 0.832031 | 0.843750 | 0.000001 | 7.867813e-06 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 0 | Person_3 |
| 4 | 419.918152 | 205.161270 | 113.457031 | 113.457016 | 0.997954 | 436.537842 | 436.316223 | 436.427032 | 437.424225 | 439.529388 | ... | 0.017944 | 0.005066 | 0.006500 | 0.000002 | 8.307397e-07 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 0 | Person_4 |
5 rows × 2182 columns
1.7 Working with multiple images¶
Detectorv1 is also flexible enough to process multiple image files if .detect() is passed a list of images. By default images will be processed serially, but you can set batch_size > 1 to process multiple images in a batch and speed up processing. NOTE: All images in a batch must have the same dimensions for batch processing. This is because behind the scenes, Detectorv1 is assembling a tensor by stacking images together. You can ask Detectorv1 to rescale images by padding and preserving proportions using the output_size in conjunction with batch_size. For example, the following would process a list of images in batches of 5 images at a time resizing each so one axis is 512:
detector_v2.detect(img_list, batch_size=5, output_size=512) # without output_size this would raise an error if image sizes differ!
In the example below we keep things simple, by process both our single and multi-face example serislly by setting batch_size = 1.
Notice how the returned Fex data class instance has 6 rows: 1 for the first face in the first image, and 5 for the faces in the second image:
NOTE: Currently batch processing images gives slightly different AU detection results due to the way that py-feat integrates the underlying models. You can examine the degree of tolerance by checking out the results of test_detection_and_batching_with_diff_img_sizes in our test-suite
img_list = [single_face_img_path, multi_face_image_path]
mixed_prediction = detector_v2.detect(img_list, batch_size=1, data_type="image")
mixed_prediction
0%| | 0/2 [00:00<?, ?it/s] 100%|██████████| 2/2 [00:00<00:00, 34.27it/s]
| FaceRectX | FaceRectY | FaceRectWidth | FaceRectHeight | FaceScore | x_0 | x_1 | x_2 | x_3 | x_4 | ... | mouthStretchRight | mouthUpperUpLeft | mouthUpperUpRight | noseSneerLeft | noseSneerRight | FrameHeight | FrameWidth | input | frame | Identity | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 129.789062 | 118.849884 | 301.164368 | 301.164368 | 0.999925 | 191.551285 | 193.610031 | 195.374664 | 198.903931 | 204.197845 | ... | 0.016357 | 0.710938 | 0.632812 | 0.000002 | 3.084540e-06 | 562.0 | 572.0 | /home/ljchang/Github/py-feat/feat/tests/data/single_face.jpg | 0 | Person_0 |
| 1 | 656.876831 | 277.657898 | 158.751953 | 158.751953 | 0.999982 | 684.472412 | 683.852234 | 683.387146 | 683.697205 | 685.402588 | ... | 0.049561 | 0.832031 | 0.843750 | 0.000002 | 4.500151e-06 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 1 | Person_1 |
| 2 | 505.674805 | 297.692688 | 149.074585 | 149.074585 | 0.999973 | 536.101135 | 535.082092 | 534.208618 | 534.063049 | 534.790955 | ... | 0.221680 | 0.792969 | 0.761719 | 0.000004 | 2.905726e-06 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 1 | Person_2 |
| 3 | 288.728638 | 222.224716 | 149.330963 | 149.330948 | 0.999939 | 313.957397 | 314.686554 | 316.436523 | 319.061493 | 322.998932 | ... | 0.034668 | 0.703125 | 0.707031 | 0.000002 | 3.963709e-06 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 1 | Person_3 |
| 4 | 197.528900 | 57.485100 | 130.303192 | 130.303162 | 0.999205 | 217.761520 | 216.361786 | 215.471039 | 215.089294 | 215.852783 | ... | 0.114258 | 0.832031 | 0.843750 | 0.000001 | 7.867813e-06 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 1 | Person_4 |
| 5 | 419.918152 | 205.161270 | 113.457031 | 113.457016 | 0.997954 | 436.537842 | 436.316223 | 436.427032 | 437.424225 | 439.529388 | ... | 0.017944 | 0.005066 | 0.006500 | 0.000002 | 8.307397e-07 | 667.0 | 1000.0 | /home/ljchang/Github/py-feat/feat/tests/data/multi_face.jpg | 1 | Person_5 |
6 rows × 2182 columns
Calling .plot_detections() will now plot detections for all images the detector was passed:
However, it's easy to use pandas slicing syntax to just grab predictions for the image you want. For example you can use .loc and chain it to .plot_detections():
# Just plot the detection corresponding to the first row in the Fex data
_figs = mixed_prediction.loc[0].plot_detections()
_figs[0]
Likewise you can use .query() and chain it to .plot_detections(). Fex data classes store each file path in the 'input' column. So we can use regular pandas methods like .unique() to get all the unique images (2 in our case) and pick the second one.
# Choose plot based on image file name
img_name = mixed_prediction["input"].unique()[1]
axes = mixed_prediction.query("input == @img_name").plot_detections()
axes[0]
Using the modular Detectorv1¶
Before Detectorv2, Py-Feat used Detectorv1 — a modular pipeline that glues together a separate pre-trained model per sub-task (face, landmarks, Action Units, emotion, head pose, identity). Reach for it when you want to swap or disable a specific model (e.g. Detectorv1(emotion_model='svm')) or need the classic modular behavior. It exposes the same .detect() API and returns the same kind of Fex object, so everything above works with either detector.
Detectorv2 is the recommended default for new work; see the two-detector overview for a full comparison.
from feat import Detectorv1
# The modular Detectorv1. Swap individual models via kwargs, e.g.
# Detectorv1(emotion_model='svm'). device was selected above (cuda/mps/cpu).
detector = Detectorv1(device=device)
/tmp/marimo_1053920/__marimo__cell_LJZf_.py:5: UserWarning: face_model='retinaface' does not regress 6DoF head pose. Pose columns are populated via the landmarks-to-pose MLP (distilled from img2pose on CelebV-HQ, ~5° avg MAE vs img2pose). Pose stays NaN if the MLP weights aren't available. Use face_model='img2pose' for the slowest, highest-accuracy path. See feat.utils.face_pose_mlp for details. detector = Detectorv1(device=device)
AU-projection visualization¶
By default .plot_detections() will overlay facial lines on top of the input image. However, it's also possible to visualize a face using Py-Feat's standardized AU landmark model, which takes the detected AUs and projects them onto a template face. You can control this by setting faces='aus' instead of the default faces='landmarks'. For more details about this kind of visualization see the visualizing facial expressions tutorial:
# AU-projection visualization (faces='aus') uses Detectorv1's named xgb
# AU model and its trained landmark viz model; Detectorv2's AUs have no
# projection model, so we use the legacy detector here. See tutorial 03 for
# more on AU visualization.
_v1_fex = detector.detect(single_face_img_path, data_type="image")
_figs = _v1_fex.plot_detections(faces='aus', muscles=True)
_figs[0]
0%| | 0/1 [00:00<?, ?it/s] 0%| | 0/1 [00:00<?, ?it/s][A 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 2.02it/s] 100%|██████████| 1/1 [00:00<00:00, 2.02it/s] Ignoring fixed y limits to fulfill fixed data aspect with adjustable data limits. Ignoring fixed y limits to fulfill fixed data aspect with adjustable data limits.
Interactive Plotting¶
You can also use the .iplot_detections() method to generate an interactive plotly figure that lets you interactively enable/disable various detector outputs:
# Interactive plotting uses the v1 detector here: Detectorv2's emotion
# columns (Neutral/Happy/...) aren't yet wired into iplot_detections.
_v1_fex = detector.detect(single_face_img_path, data_type="image")
_v1_fex.iplot_detections(bounding_boxes=True, emotions=True)
0%| | 0/1 [00:00<?, ?it/s] 0%| | 0/1 [00:00<?, ?it/s][A 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 16.56it/s]