region_maps

feat.utils.region_maps

Facial AU / ARKit-blendshape region overlays on the MediaPipe-478 mesh.

This is the dense-mesh, non-overlapping successor to the dlib-68 muscle polygons in feat.plotting.draw_muscles and the overlapping muscle map in feat.utils.muscle_to_landmark. It ships two maps:

• au_region_map.json — the 20 FACS AUs, left+right merged per AU.
• blendshape_region_map.json — the spatially-distinct ARKit blendshapes, Left/Right kept independent (ARKit pre-splits ...Left/...Right).

Both are built by a winner-take-all geodesic-Voronoi partition of the mesh: each vertex is assigned to the single nearest region seed by walking the mesh surface (Dijkstra over the triangle-edge graph, capped at TIGHTNESS of the face span), with the eye/mouth aperture rings walling growth off. The partition is non-overlapping by construction, which fixes the heavy region overlap of the geodesic-grow muscle map (where 323/394 covered verts sat in >=2 muscles).

The AU <-> muscle <-> blendshape correspondence is grounded in: * FACS (Ekman & Friesen) — AU -> muscle. * Melinda Ozel's ARKit-to-FACS cheat sheet (melindaozel.com/arkit-to-facs-cheat-sheet). * pooyadeperson's "Ultimate Guide to ARKit's 52 Facial Blendshapes" (anatomy refs).

Rendering smooths the coarse 468-vertex mesh by midpoint-subdividing it SUBDIV_LEVELS times before filling triangles. The subdivision topology and the dense per-vertex region labels are fixed in index space, so the same assets overlay a live detected 478-mesh — only the dense vertex positions are recomputed per frame (dense_positions). See feat.plotting.plot_face_regions.

build_au_region_map()

{AU: {muscles, mp478_vertices, n_vertices}} — muscle partition merged to AU (left+right share an AU).

Source code in feat/utils/region_maps.py

def build_au_region_map():
    """``{AU: {muscles, mp478_vertices, n_vertices}}`` — muscle partition merged
    to AU (left+right share an AU)."""
    seeds = {m: s for m, (au, s) in MUSCLE_SEEDS.items()}
    label, _, _ = _build_partition_labels(seeds)
    muscle_au = {m: au for m, (au, s) in MUSCLE_SEEDS.items()}
    by_au: dict[str, set] = defaultdict(set)
    muscles_by_au: dict[str, set] = defaultdict(set)
    for v, muscles in label.items():
        for muscle in muscles:
            by_au[muscle_au[muscle]].add(v)
            muscles_by_au[muscle_au[muscle]].add(muscle)
    out = {}
    for au in sorted(by_au):
        verts = sorted(by_au[au])
        out[au] = dict(muscles=sorted(muscles_by_au[au]),
                       mp478_vertices=verts, n_vertices=len(verts))
    return out

build_blendshape_region_map()

{blendshape: {au, muscle, side, mp478_vertices, n_vertices}}.

L/R pairs share one symmetric mesh FEATURE (their seeds merged) grown by the geodesic-Voronoi partition; each sided blendshape is then the half of that feature on its side of the facial midline (center shapes keep the whole, bilateral feature). So L/R divide cleanly down the middle — no winner-take- all asymmetry, no shared midline seam. Non-overlapping by construction.

Source code in feat/utils/region_maps.py

def build_blendshape_region_map():
    """``{blendshape: {au, muscle, side, mp478_vertices, n_vertices}}``.

    L/R pairs share one symmetric mesh FEATURE (their seeds merged) grown by the
    geodesic-Voronoi partition; each sided blendshape is then the half of that
    feature on its side of the facial midline (center shapes keep the whole,
    bilateral feature). So L/R divide cleanly down the middle — no winner-take-
    all asymmetry, no shared midline seam. Non-overlapping by construction."""
    label, V, _ = _build_partition_labels(_feature_seeds())
    feat_verts: dict[str, set] = defaultdict(set)
    for v, names in label.items():
        for f in names:
            feat_verts[f].add(v)
    _, _, axis = _mirror_map(np.asarray(V, dtype=np.float64))
    eps = 0.01 * float(V[:, 0].max() - V[:, 0].min())
    out = {}
    for bs, (au, muscle, side, _seeds) in BLENDSHAPE_SEEDS.items():
        fv = feat_verts[_feature_of(bs)]
        if side == "L":          # subject-left half (x > midline)
            verts = sorted(v for v in fv if V[v, 0] > axis + eps)
        elif side == "R":        # subject-right half (x < midline)
            verts = sorted(v for v in fv if V[v, 0] < axis - eps)
        else:                    # center: whole bilateral feature
            verts = sorted(fv)
        out[bs] = dict(au=au, muscle=muscle, side=side,
                       mp478_vertices=verts, n_vertices=len(verts))
    return out

dense_positions(V, parents)

Rebuild subdivided vertex positions for an arbitrary base mesh V (e.g. a detected 478-mesh) given the parents list from subdivide. V supplies the original vertices; midpoints are filled by averaging.

Source code in feat/utils/region_maps.py

def dense_positions(V, parents):
    """Rebuild subdivided vertex positions for an arbitrary base mesh ``V``
    (e.g. a detected 478-mesh) given the ``parents`` list from ``subdivide``.
    ``V`` supplies the original vertices; midpoints are filled by averaging."""
    pos = [np.asarray(p, dtype=np.float64) for p in V]
    for a, b in parents:
        pos.append((pos[a] + pos[b]) / 2.0)
    return np.array(pos)

geodesic_voronoi(seeds_by_region, adj, xy, aperture, max_geo)

Winner-take-all surface partition: assign each vertex to the nearest seed's region by Dijkstra over the edge graph (euclidean edge weights on xy), capped at max_geo, never crossing aperture verts.

Returns {vertex_index: region_name}. Non-overlapping by construction — a vertex is claimed once, by whichever region reaches it first/closest.

Source code in feat/utils/region_maps.py

def geodesic_voronoi(seeds_by_region, adj, xy, aperture, max_geo):
    """Winner-take-all surface partition: assign each vertex to the nearest
    seed's region by Dijkstra over the edge graph (euclidean edge weights on
    ``xy``), capped at ``max_geo``, never crossing ``aperture`` verts.

    Returns ``{vertex_index: region_name}``. Non-overlapping by construction —
    a vertex is claimed once, by whichever region reaches it first/closest."""
    label: dict[int, str] = {}
    dist: dict[int, float] = {}
    pq: list = []
    for region, seeds in seeds_by_region.items():
        for s in seeds:
            if s in aperture or s >= len(xy):
                continue
            heapq.heappush(pq, (0.0, int(s), region))
    while pq:
        d, v, region = heapq.heappop(pq)
        if v in label and dist.get(v, np.inf) <= d:
            continue
        label[v] = region
        dist[v] = d
        for w in adj.get(v, []):
            if w in aperture:
                continue
            nd = d + float(np.linalg.norm(xy[v] - xy[w]))
            if nd <= max_geo and (w not in dist or nd < dist[w]):
                dist[w] = nd
                heapq.heappush(pq, (nd, w, region))
    return label

load_au_region_map()

Bundled non-overlapping AU region map (20 AUs, L+R merged). Each value: {"muscles": [...], "mp478_vertices": [...], "n_vertices": int}.

Source code in feat/utils/region_maps.py

def load_au_region_map() -> dict:
    """Bundled non-overlapping AU region map (20 AUs, L+R merged). Each value:
    ``{"muscles": [...], "mp478_vertices": [...], "n_vertices": int}``."""
    return _load(AU_MAP_FILENAME)

load_blendshape_region_map()

Bundled non-overlapping blendshape region map (L/R independent). Each value: {"au", "muscle", "side", "mp478_vertices", "n_vertices"}.

Source code in feat/utils/region_maps.py

def load_blendshape_region_map() -> dict:
    """Bundled non-overlapping blendshape region map (L/R independent). Each
    value: ``{"au", "muscle", "side", "mp478_vertices", "n_vertices"}``."""
    return _load(BLENDSHAPE_MAP_FILENAME)

project_xy(V)

Frontal x/y projection, flipped so the forehead (v10) is above the chin (v152). V may be the canonical mesh or a detected mesh (N>=153, x/y in cols 0/1). Used both for geodesic distances and for plotting.

Source code in feat/utils/region_maps.py

def project_xy(V: np.ndarray) -> np.ndarray:
    """Frontal x/y projection, flipped so the forehead (v10) is above the chin
    (v152). ``V`` may be the canonical mesh or a detected mesh (N>=153, x/y in
    cols 0/1). Used both for geodesic distances and for plotting."""
    xy = np.asarray(V, dtype=np.float64)[:, :2].copy()
    if xy[10, 1] < xy[152, 1]:
        xy[:, 1] = -xy[:, 1]
    return xy

render_assets(kind='au')

Cached rendering assets for kind in {"au", "blendshape"}:

{"parents": [(a,b)...], "tris": dense_tris[K,3], "region_verts": {region: set(dense vert ids)}, "axis": float, "n_base": 468}

Regions are resolved on the SUBDIVIDED canonical mesh (smooth boundaries), keyed by AU for "au" and by FEATURE (L/R merged) for "blendshape" — the renderer splits a feature into Left/Right by the midline axis using triangle centroids. parents/tris/region_verts are fixed in index space, so a deformed 478-mesh just needs dense_positions(mesh, parents).

Source code in feat/utils/region_maps.py

def render_assets(kind: str = "au") -> dict:
    """Cached rendering assets for ``kind`` in {"au", "blendshape"}:

    ``{"parents": [(a,b)...], "tris": dense_tris[K,3],
       "region_verts": {region: set(dense vert ids)}, "axis": float, "n_base": 468}``

    Regions are resolved on the SUBDIVIDED canonical mesh (smooth boundaries),
    keyed by AU for ``"au"`` and by FEATURE (L/R merged) for ``"blendshape"`` —
    the renderer splits a feature into Left/Right by the midline ``axis`` using
    triangle centroids. ``parents``/``tris``/``region_verts`` are fixed in index
    space, so a deformed 478-mesh just needs ``dense_positions(mesh, parents)``.
    """
    if kind not in _RENDER_ASSETS:
        if kind not in ("au", "blendshape"):
            raise ValueError(f"kind must be 'au' or 'blendshape', got {kind!r}")

        V, tris = _canonical_geometry()
        n_base = len(V)
        aperture = {a for a in APERTURE if a < n_base}
        Vd, tris_d, ap_d, parents = subdivide(V, tris, aperture, SUBDIV_LEVELS)
        xy = project_xy(Vd)
        adj = build_adjacency(tris_d, len(Vd))
        face = float(np.linalg.norm(xy.max(0) - xy.min(0)))
        max_geo = face * TIGHTNESS

        if kind == "au":
            seeds = {m: s for m, (au, s) in MUSCLE_SEEDS.items()}
            muscle_au = {m: au for m, (au, s) in MUSCLE_SEEDS.items()}
            raw = geodesic_voronoi(seeds, adj, xy, ap_d, max_geo)
            label = {v: muscle_au[m] for v, m in raw.items()}
        else:
            label = geodesic_voronoi(_feature_seeds(), adj, xy, ap_d, max_geo)
        label = _symmetrize_partition(label, Vd)
        region_verts: dict[str, set] = defaultdict(set)
        for v, names in label.items():
            for name in names:
                region_verts[name].add(v)

        _RENDER_ASSETS[kind] = dict(parents=parents, tris=tris_d,
                                    region_verts=dict(region_verts),
                                    axis=float(_mirror_map(Vd)[2]), n_base=n_base)
    a = _RENDER_ASSETS[kind]
    return dict(parents=list(a["parents"]), tris=a["tris"],
                region_verts={k: set(v) for k, v in a["region_verts"].items()},
                axis=a["axis"], n_base=a["n_base"])

subdivide(V, tris, aperture, levels)

levels rounds of 1-to-4 midpoint subdivision. Original vertices keep their indices (so seeds and stored vertex ids stay valid). Returns (V_dense, tris_dense, aperture_dense, parents) where parents lists, in creation order, the (a, b) parent pair of each appended midpoint — so a deformed mesh's dense positions can be rebuilt with dense_positions. A midpoint inherits aperture status only if BOTH parents are aperture.

Source code in feat/utils/region_maps.py

def subdivide(V, tris, aperture, levels):
    """``levels`` rounds of 1-to-4 midpoint subdivision. Original vertices keep
    their indices (so seeds and stored vertex ids stay valid). Returns
    ``(V_dense, tris_dense, aperture_dense, parents)`` where ``parents`` lists,
    in creation order, the ``(a, b)`` parent pair of each appended midpoint — so
    a deformed mesh's dense positions can be rebuilt with ``dense_positions``.
    A midpoint inherits aperture status only if BOTH parents are aperture."""
    V = [np.asarray(p, dtype=np.float64) for p in V]
    ap = set(aperture)
    parents: list[tuple[int, int]] = []
    for _ in range(levels):
        cache: dict[tuple[int, int], int] = {}
        new_tris = []

        def mid(a, b):
            k = (a, b) if a < b else (b, a)
            idx = cache.get(k)
            if idx is None:
                idx = len(V)
                cache[k] = idx
                V.append((V[a] + V[b]) / 2.0)
                parents.append((a, b))
                if a in ap and b in ap:
                    ap.add(idx)
            return idx

        for a, b, c in tris:
            a, b, c = int(a), int(b), int(c)
            ab, bc, ca = mid(a, b), mid(b, c), mid(c, a)
            new_tris += [(a, ab, ca), (ab, b, bc), (ca, bc, c), (ab, bc, ca)]
        tris = np.array(new_tris, dtype=int)
    return np.array(V), tris, ap, parents