willowtreeapps / willowtreeapps/vocable-android

Spike: prototype MediaPipe Face Landmarker as ARCore replacement for head tracking

Open
#676 1 comment 0 reactions 1 assignee View on GitHub

@MicGon7 is already working on this.

Since Aug 7, 2026.

Dominant language
Kotlin
Stars
127
Forks
21
Avg merge
16m
Merged PRs (30d)
4

Description

Why

#629's research into head-tracking tuning surfaced a bigger question than smoothing math: the two real shipped apps solving this exact problem for this population (Eva Facial Mouse, Google Project Gameface) don't use iOS's PID-controller approach at all, and more importantly, Project Gameface — Google's own current reference implementation for accessibility cursor control — doesn't use ARCore Augmented Faces, the API Vocable Android is built on. It uses MediaPipe Face Landmarker instead: richer data (468 landmarks + 52 blendshapes vs. ARCore's single nose-tip pose), no ARCore device-support gate, free/open-source, actively maintained by Google for this exact use case. Before investing further in tuning the current ARCore-based smoothing, we should validate whether swapping the tracking backend is viable and worth the larger migration.

Scope

  • Build a MediaPipe-based tracking path behind a debug flag, side-by-side with the current ARCore path — this validates feasibility and feel, not a hardened production swap.
  • CameraX + MediaPipe FaceLandmarker feeding the existing FaceTrackingViewModel-shaped vector output, unchanged downstream (convertCoordSystems, GazePointer, GazeInteractionManager, dwell-click all stay as-is).
  • Side-by-side toggle so the MediaPipe path and current ARCore path can be compared on the same device.
  • If it proves out, this branch becomes the basis for the production migration (tracked as a separate follow-up issue), not a throwaway to be discarded.

Acceptance Criteria

  • Prototype compiles and runs on ≥2 real devices (not emulator-only — front camera + live-stream face tracking needs real hardware)
  • Toggle exists to switch between MediaPipe and ARCore tracking paths for comparison
  • Informal feel/latency/jitter assessment recorded (doesn't need to be rigorous, just a documented judgment call)
  • Findings (worth pursuing further vs. not) reported back on parent #629

Deferred (only if we proceed)

  • Removing the ARCore path / FaceTrackingManager rewiring
  • Blendshape-driven gestures (blink-to-click, etc.)
  • Test coverage, work-log doc

Links

Part of #629

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.