benaclejames / benaclejames/VRCFaceTracking

[RFC] Expose standardized OSC interface for raw tracking data

Open
#128 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
907
Forks
135
Avg merge
11h 28m
Merged PRs (30d)
7

Description

In this RFC I propose a unified interface for submitting raw tracking data, for sources like ALXR, ALVR, VRCFTQuestPro and other projects.

## Description

The 2 main face tracking platforms supported today are HTC eye and lip trackers, and the Quest Pro. Another source is the combined eye gaze exposed by the OpenXR extension `XR_EXT_eye_gaze_interaction`, which, among others, is supported by the Pico platform.

For the protocol I propose OSC ovr UDP.

The addresses of the OSC inputs are grouped by scope. For illustration here I divide each address into prefix and suffix, which should be used together to form a single path.

I propose 4 prefixes: `/tracking/eye`, `/tracking/face_fb`, `/tracking/eye_htc`, `/tracking/lip_htc`.

`/tracking/eye` contains unopinionated inputs relative to the eyeballs movements, not constrained to any specific vendor. This prefix can have `/center`, `/left` and `/right` subpaths. An example for a full path is: `/tracking/eye/left/pitch_yaw`. These are the proposed suffixes:

| Suffix | OSC args |
| ------------------- | ---------------- |
| `/active` | `Bool` |
| `/pitch_yaw` | `[Float, Float]` |
| `/vec` | `[Float, Float, Float]` |
| `/quat` | `[Float, Float, Float, Float]` |

Clients can send any of these inputs at any time and VRCFaceTracking should adapt to read any of these inputs in any order. When a client detects that an eye is no longer actively tracked, it should send `False` to the corresponding `/active` input as soon as possible; while it's not mandatory to send `/active = True` when the corresponding input is available, due to the unreliable nature of UDP, `/active = False` inputs should be sent periodically. If VRCFaceTracking does not receive any input for a particular eye for more than 10 seconds then it is free to perform any suitable idle animation.

Pitch and yaw are in radians, where 0 pitch and 0 yaw corresponds to the eye looking in the forward direction. The rotations follow the right hand rule, so +pitch is up and +yaw is left. `/vec` inputs are in order X, Y, and Z of the forward gaze direction. `/quat` inputs are in order W, X, Y, Z of the forward gaze direction. All orientation conventions should be local relative to the head orientation.

These are the suffixes relative to the `/tracking/face_fb` prefix:

| Suffix | OSC args |
| --------------- | -------- |
| `/brow_lowerer_l` | `Float` |
| `/brow_lowerer_r` | `Float` |
| `/cheeck_puff_l` | `Float` |
| ... | ... |

All other suffixes are extrapolated from the [`XR_FB_face_tracking` OpenXR extension](https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#_conventions_of_blend_shapes) by transforming constants to snake_case.

At the address `/tracking/face_fb` (full path), clients can send the full vector of 63 floats in one go.

Suffixes for `/tracking/eye_htc` are extracted in the same way from [`XR_HTC_facial_tracking`](https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#XR_HTC_facial_tracking):

| Suffix | OSC args |
| ------------- | -------- |
| `/left_blink` | `Float` |
| `/left_wide` | `Float` |
| `/right_blink` | `Float` |
| ... | ... |

And for `/tracking/lip_htc`:

| Suffix | OSC args |
| ------------- | -------- |
| `/jaw_right` | `Float` |
| `/jaw_left` | `Float` |
| `/jaw_forward` | `Float` |
| ... | ... |

VRCFaceTracking is responsible to handle any combination of inputs from different sources/clients. In the most common case FB and HTC inputs will not be mixed.

Similarly for fb, clients can send full parameters vectors at `/tracking/eye_htc` and `/tracking/lip_htc`.

## Rationale

> Why OSC over UDP?

OSC is a well established protocol in the VRChat community, with a simple interface. UDP is suitable for low latency transmission, and the integrity of the transmitted data is non-critical.

> Why duplicating FB and HTC inputs?

The goal of this interface is to be as little opinionated as possible. These platforms have different conventions and so they should have separate input endpoints. The data should be pushed to the interface with as little preprocessing as possible, since it's VRCFaceTracking job to pack the data in a suitable streaming format for VRChat. The only exception is for FB eye tracking which is sent in a pose form, with global reference frame, which should be converted to local relative to the head orientation.

## Possible extension

This interface should support exposing the internal UnifiedExpressions as OSC inputs directly. The UnifiedExpressions set is in the process of being stabilized.

This interface could also be extended for hand tracking input, using path prefixes `/tracking/hand/left` and `/tracking/hand/right`. The suffixes can be:

| Suffix | OSC args |
| ------------- | ---------------- |
| `/active` | `Bool` |
| `/thumb_curl` | `Float` |
| `/index_curl` | `Float` |
| `/middle_curl` | `Float` |
| `/ring_curl` | `Float` |
| `/pinky_curl` | `Float` |
| `/splay` | `Float` |
| `/thumb_rot_xy` | `[Float, Float]` |

`/splay` is average splay of the fingers from index to pinky.
`/thumb_rot_xy` convention is to be defined.

------------------

EDIT: Changed `Enabled` -> `Active`, refactored eye paths.
EDIT2: Added OSC batch inputs for face_fb, eye_htc and lip_htc.
EDIT3: Support for UnifiedExpressions inputs.
EDIT4: Use snake_case path suffixes for consistency.

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository files, tests, or entry points are named. Start by resolving the proposed OSC-over-UDP paths and OpenXR-derived conventions; done is an agreed interface specification, since this RFC does not identify implementation or test locations.

Written by the indexing model from the issue text.

Assessment

Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.