willowtreeapps / willowtreeapps/vocable-android
Spike: prototype MediaPipe Face Landmarker as ARCore replacement for head tracking
@MicGon7 is already working on this.
Since Aug 7, 2026.
- Dominant language
- Kotlin
- Stars
- 127
- Forks
- 21
- Avg merge
- 16m
- Merged PRs (30d)
- 4
Description
Why
#629's research into head-tracking tuning surfaced a bigger question than smoothing math: the two real shipped apps solving this exact problem for this population (Eva Facial Mouse, Google Project Gameface) don't use iOS's PID-controller approach at all, and more importantly, Project Gameface — Google's own current reference implementation for accessibility cursor control — doesn't use ARCore Augmented Faces, the API Vocable Android is built on. It uses MediaPipe Face Landmarker instead: richer data (468 landmarks + 52 blendshapes vs. ARCore's single nose-tip pose), no ARCore device-support gate, free/open-source, actively maintained by Google for this exact use case. Before investing further in tuning the current ARCore-based smoothing, we should validate whether swapping the tracking backend is viable and worth the larger migration.
Scope
- Build a MediaPipe-based tracking path behind a debug flag, side-by-side with the current ARCore path — this validates feasibility and feel, not a hardened production swap.
- CameraX + MediaPipe
FaceLandmarkerfeeding the existingFaceTrackingViewModel-shaped vector output, unchanged downstream (convertCoordSystems,GazePointer,GazeInteractionManager, dwell-click all stay as-is). - Side-by-side toggle so the MediaPipe path and current ARCore path can be compared on the same device.
- If it proves out, this branch becomes the basis for the production migration (tracked as a separate follow-up issue), not a throwaway to be discarded.
Acceptance Criteria
- Prototype compiles and runs on ≥2 real devices (not emulator-only — front camera + live-stream face tracking needs real hardware)
- Toggle exists to switch between MediaPipe and ARCore tracking paths for comparison
- Informal feel/latency/jitter assessment recorded (doesn't need to be rigorous, just a documented judgment call)
- Findings (worth pursuing further vs. not) reported back on parent #629
Deferred (only if we proceed)
- Removing the ARCore path /
FaceTrackingManagerrewiring - Blendshape-driven gestures (blink-to-click, etc.)
- Test coverage, work-log doc
Links
Part of #629
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.