android / android/android-xr-unity-package

Face tracking returns all-zero blendshape data despite FACE_TRACKING permission granted

Open
#22 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
94
Forks
17
PR merge metrics
No merged PRs in 30d

Description

## Summary

After granting `android.permission.FACE_TRACKING`, the `XR_ANDROID_face_tracking` subsystem reports as available but returns all-zero blendshape values. Over a 1.5-second sampling window, 0 out of 87 frames contained any blendshape value above 0.001.

## What We Observe

- `XRFaceTrackingFeature` reports the subsystem as available
- All 10 blendshape weights we read (brow raiser L/R, lip corner puller L/R, cheek puff L/R, eyes closed L/R, lip pucker L/R) return 0.0 every frame
- The user has previously calibrated face tracking via Settings > Avatars > Calibrate on the headset
- The permission `android.permission.FACE_TRACKING` is granted (confirmed via `dumpsys package`)

## Questions

1. **Does face calibration (Settings > Avatars > Calibrate) persist across app launches and headset reboots?** We calibrated once and expected it to persist, but subsequent sessions return zeros.
2. **Is there a way to trigger face calibration from within the app**, similar to how eye calibration can be launched via intent (`com.samsung.fitting.eyetracking.calibration`)?
3. **Is there a warm-up period or initialization sequence required** before blendshape data starts flowing? Our 1.5-second warm-up window may not be long enough.
4. **Does the face tracking subsystem require the headset to be in a specific mode** (e.g., passthrough enabled) to return data?

## Use Case

We are building a clinical neurologic exam (SDNE) that measures facial asymmetry — comparing left vs. right blendshape peaks during prompted expressions (raise eyebrows, smile, puff cheeks). This is a standard component of neurologic assessment (facial nerve CN VII evaluation).

## Environment

- Device: Samsung Galaxy XR (SM_I610)
- OS: Android XR
- Unity: 6000.3.6f1
- OpenXR runtime with `XR_ANDROID_face_tracking`
- Permission granted: `android.permission.FACE_TRACKING`
- Blendshapes accessed via `XRFaceTrackingFeature` / OpenXR face tracking extension

Contributor guide

Open the contributing guide

Research direction

Start with XRFaceTrackingFeature and the XR_ANDROID_face_tracking integration, reproducing the all-zero blendshape results on the listed Samsung Galaxy XR environment. Check the permission, calibration, initialization, warm-up, and headset-mode conditions raised in the issue; done means identifying the runtime requirement or confirming a package/runtime defect and recording the verified behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, csharp, unity
Domain
computer-vision, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.