alvr-org / alvr-org/ALVR

Support visionOS Foveated Streaming

未关闭
#3,206 9 条评论 18 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Rust
星标
7.9k
派生
708
平均合并
3 天 23 小时
30 天内合并 PR
11

描述

Apple just dropped the [Foveated Streaming](https://developer.apple.com/documentation/foveatedstreaming) framework today (visionOS 26.4 beta) along with a [reference implementation](https://github.com/apple/StreamingSession). It's a first-party API that gives streaming apps access to gaze-directed foveation data — the thing that's been blocked by Apple until now.

Directly relevant to #20, #133, and #157. As @shinyquagsire23 noted in #133: "Eye tracking is an Apple limitation." This could be the answer.

## What the framework gives you

The whole thing is built around a session management protocol that's transport-agnostic:

- mDNS discovery via `_apple-foveated-streaming._tcp`
- TCP + JSON messaging for connection lifecycle and QR code pairing
- Session states: `WAITING` → `CONNECTING` → `CONNECTED` → `PAUSED` → `DISCONNECTED`
- Auto pause/resume on headset removal

On the visionOS side:
- `FoveatedStreamingSession` manages the connection to the streaming endpoint
- `FoveatedStreamingSpace` is a new `ImmersiveSpace` variant with native foveation support
- Bidirectional message channel for custom data exchange between client and endpoint
- Supports progressive and mixed immersion styles

The endpoint receives approximate gaze region data and renders high-res content only in that region, reducing bandwidth and compute while improving perceived quality.

## How this maps to ALVR's current architecture

After some analysis, it seems like here's where things stand and what would need to change:

**Where ALVR is now:**
- [`FFR.swift`](https://github.com/alvr-org/alvr-visionos/blob/cbf6371f8f109c765603443ee7631fd90de74ee3/ALVRClient/FFR.swift#L10-L27) does fixed foveated rendering with static center coordinates from server config (`centerSizeX/Y`, `centerShiftX/Y`, `edgeRatioX/Y`). Fovea is always center-of-FOV.
- [`RealityKitEyeTrackingSystem.swift`](https://github.com/alvr-org/alvr-visionos/blob/cbf6371f8f109c765603443ee7631fd90de74ee3/ALVRClient/RealityKitEyeTrackingSystem.swift#L1-L60) is the side-channel workaround — screen recording broadcast extensions + CFNotificationCenter shift registers to get approximate eye position. Requires the RealityKit renderer, limited precision.
- [`EventHandler.swift`](https://github.com/alvr-org/alvr-visionos/blob/cbf6371f8f109c765603443ee7631fd90de74ee3/ALVRClient/EventHandler.swift#L314-L352) handles mDNS/Bonjour discovery and connection management.
- [`ALVRClientApp.swift`](https://github.com/alvr-org/alvr-visionos/blob/cbf6371f8f109c765603443ee7631fd90de74ee3/ALVRClient/ALVRClientApp.swift#L200-L222) defines three immersive spaces (DummyImmersiveSpace, RealityKitClient, MetalClient) via `CompositorLayer` / `CompositorServices`.

**Proposed integration (hybrid approach — keeps ALVR's transport):**

1. Add a `FoveatedStreamingSpace` alongside the [existing immersive spaces](https://github.com/alvr-org/alvr-visionos/blob/cbf6371f8f109c765603443ee7631fd90de74ee3/ALVRClient/ALVRClientApp.swift#L200-L222) in `ALVRClientApp.swift` as a new rendering mode.

2. Implement the session management protocol on the streamer side. It's straightforward TCP + JSON, similar to what ALVR already does for its own connection management. The mDNS service type changes to `_apple-foveated-streaming._tcp` and the handshake adds QR code pairing.

3. Use Apple's gaze region data to drive [`FFR.swift`](https://github.com/alvr-org/alvr-visionos/blob/cbf6371f8f109c765603443ee7631fd90de74ee3/ALVRClient/FFR.swift#L10-L27) — dynamically update `centerShiftX/Y` from the framework's gaze callbacks instead of static values. This replaces the [`RealityKitEyeTrackingSystem`](https://github.com/alvr-org/alvr-visionos/blob/cbf6371f8f109c765603443ee7631fd90de74ee3/ALVRClient/RealityKitEyeTrackingSystem.swift) workaround entirely.

4. Keep ALVR's video transport (H.264/HEVC/AV1 pipeline). The framework uses NVIDIA CloudXR as its reference transport, but the session management layer is independent. ALVR can implement the session protocol for foveation data while keeping its own streaming pipeline.

There's also the option of going full CloudXR, but that would drop ALVR's cross-platform support and require NVIDIA GPUs. Probably only interesting as a niche mode for users with the right hardware.

**Client-side changes (alvr-visionos):**
- New immersive space type wrapping `FoveatedStreamingSpace`
- `FoveatedStreamingSession` connection management alongside existing ALVR connection
- Deprecate [`RealityKitEyeTrackingSystem.swift`](https://github.com/alvr-org/alvr-visionos/blob/cbf6371f8f109c765603443ee7631fd90de74ee3/ALVRClient/RealityKitEyeTrackingSystem.swift) broadcast extension workaround
- Forward gaze region data to server via ALVR's existing data channel

**Server-side changes (ALVR streamer):**
- Implement Apple's [session management protocol](https://developer.apple.com/documentation/foveatedstreaming/establishing-foveated-streaming-sessions-with-apple-vision-pro) (mDNS + TCP/JSON)
- Accept dynamic foveation center coordinates from the client
- Update encoder foveation parameters per-frame based on gaze data

Worth noting: Apple's sample repo also includes a `StreamingSession.xcframework` for iOS, so this could eventually enable ALVR streaming to iPhones/iPads too.

## References

- [Foveated Streaming framework docs](https://developer.apple.com/documentation/foveatedstreaming) (visionOS 26.4+ beta)
- [apple/StreamingSession sample code](https://github.com/apple/StreamingSession) — C# Windows app + OpenXR sample
- [Session management protocol spec](https://developer.apple.com/documentation/foveatedstreaming/establishing-foveated-streaming-sessions-with-apple-vision-pro)
- [Creating a foveated streaming client on visionOS](https://developer.apple.com/documentation/foveatedstreaming/creating-a-foveated-streaming-client-on-visionos)

## Beta status

The framework is on visionOS 26.4 developer beta, so APIs could still change before release. Probably makes sense to develop this on a feature branch and gate it behind a setting in [`GlobalSettings`](https://github.com/alvr-org/alvr-visionos/blob/cbf6371f8f109c765603443ee7631fd90de74ee3/ALVRClient/GlobalSettings.swift#L10-L37) — same pattern as `experimental40ppd`, `enablePersonaFaceTracking`, etc. That way it can ship disabled by default and get turned on once the framework goes stable.

The session protocol itself is fully documented with complete message formats and is implementable independently of CloudXR. Given how long eye-tracked foveation has been blocked (#20 is from Feb 2024), seems worth starting the work now even if it can't ship until visionOS 26.4 goes GA.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。