android / android/android-xr-unity-package
Face tracking returns all-zero blendshape data despite FACE_TRACKING permission granted
- Dominant language
- C#
- Stars
- 94
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
After granting `android.permission.FACE_TRACKING`, the `XR_ANDROID_face_tracking` subsystem reports as available but returns all-zero blendshape values. Over a 1.5-second sampling window, 0 out of 87 frames contained any blendshape value above 0.001.
## What We Observe
- `XRFaceTrackingFeature` reports the subsystem as available
- All 10 blendshape weights we read (brow raiser L/R, lip corner puller L/R, cheek puff L/R, eyes closed L/R, lip pucker L/R) return 0.0 every frame
- The user has previously calibrated face tracking via Settings > Avatars > Calibrate on the headset
- The permission `android.permission.FACE_TRACKING` is granted (confirmed via `dumpsys package`)
## Questions
1. **Does face calibration (Settings > Avatars > Calibrate) persist across app launches and headset reboots?** We calibrated once and expected it to persist, but subsequent sessions return zeros.
2. **Is there a way to trigger face calibration from within the app**, similar to how eye calibration can be launched via intent (`com.samsung.fitting.eyetracking.calibration`)?
3. **Is there a warm-up period or initialization sequence required** before blendshape data starts flowing? Our 1.5-second warm-up window may not be long enough.
4. **Does the face tracking subsystem require the headset to be in a specific mode** (e.g., passthrough enabled) to return data?
## Use Case
We are building a clinical neurologic exam (SDNE) that measures facial asymmetry — comparing left vs. right blendshape peaks during prompted expressions (raise eyebrows, smile, puff cheeks). This is a standard component of neurologic assessment (facial nerve CN VII evaluation).
## Environment
- Device: Samsung Galaxy XR (SM_I610)
- OS: Android XR
- Unity: 6000.3.6f1
- OpenXR runtime with `XR_ANDROID_face_tracking`
- Permission granted: `android.permission.FACE_TRACKING`
- Blendshapes accessed via `XRFaceTrackingFeature` / OpenXR face tracking extension
Contributor guide
Research direction
Start with XRFaceTrackingFeature and the XR_ANDROID_face_tracking integration, reproducing the all-zero blendshape results on the listed Samsung Galaxy XR environment. Check the permission, calibration, initialization, warm-up, and headset-mode conditions raised in the issue; done means identifying the runtime requirement or confirming a package/runtime defect and recording the verified behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, csharp, unity
- Domain
- computer-vision, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100