arkavo-org / arkavo-org/VRMMetalKit
ARKit: QoS controller with timestamp interpolation
- Dominant language
- Swift
- Stars
- 6
- Forks
- 2
- Avg merge
- 18h 51m
- Merged PRs (30d)
- 26
Description
## Description
Implement QoS (Quality of Service) controller to handle timestamp interpolation, jitter management, and adaptive thresholds for ARKit metadata streams. This addresses network jitter and timing variations in Continuity Camera scenarios.
## Problem Statement
Current implementation:
- Simple staleness detection (fixed 150ms threshold)
- No interpolation between frames
- No jitter handling for network transport
- Fixed priority strategies
Real-world Continuity Camera scenarios need:
- Variable network latency (USB-C vs Wi-Fi)
- Frame drops and out-of-order delivery
- Adaptive quality based on connection
- Smooth interpolation when frames are delayed
## Proposed Components
### 1. QoSController
```swift
public final class QoSController: @unchecked Sendable {
/// Connection quality assessment
public enum Quality {
case excellent // <50ms latency, <5% jitter
case good // <100ms latency, <10% jitter
case fair // <150ms latency, <20% jitter
case poor // >150ms latency or >20% jitter
}
/// Jitter buffer for smoothing
private var jitterBuffer: [(timestamp: TimeInterval, data: ARKitData)] = []
/// Track latency statistics
private var latencyHistory: [TimeInterval] = []
/// Interpolate between frames when needed
func interpolate(from: ARKitFaceBlendShapes, to: ARKitFaceBlendShapes, alpha: Float) -> ARKitFaceBlendShapes
/// Assess current connection quality
func assessQuality() -> Quality
/// Adjust staleness threshold based on quality
func adaptiveStalenesss() -> TimeInterval
}
```
### 2. Timestamp Interpolation
**Face Tracking:**
- Linear interpolation between blend shape weights
- Handle missing frames gracefully
- Extrapolate for short gaps (<50ms)
**Body Tracking:**
- SLERP for quaternion rotations (requires issue #25)
- Linear for positions
- Handle partial skeleton interpolation
### 3. Adaptive Thresholds
Adjust staleness threshold based on:
- Measured network latency
- Jitter magnitude
- Connection type (USB-C vs Wi-Fi)
- Frame drop rate
Example:
```
USB-C: 50ms threshold (low latency)
Wi-Fi good: 100ms threshold
Wi-Fi fair: 150ms threshold
Wi-Fi poor: 200ms threshold (or disable)
```
### 4. Source Manager
Centralized management of multiple sources:
```swift
public final class ARMetadataSourceManager: Sendable {
/// All registered sources
private var sources: [UUID: ARMetadataSource] = [:]
/// QoS controller per source
private var qosControllers: [UUID: QoSController] = [:]
/// Register/unregister sources
func register(_ source: ARMetadataSource)
func unregister(_ sourceID: UUID)
/// Get best source based on quality
func selectBestSource(strategy: SourcePriority) -> ARMetadataSource?
/// Get quality report for all sources
func qualityReport() -> [UUID: QoSController.Quality]
}
```
## Implementation Plan
### Phase 1: QoS Metrics (1 day)
- Latency tracking
- Jitter calculation
- Quality assessment
- Adaptive threshold logic
### Phase 2: Interpolation (2 days)
- Blend shape linear interpolation
- Skeleton SLERP (depends on #25)
- Extrapolation for short gaps
- Frame buffering
### Phase 3: Source Manager (1 day)
- Multi-source registration
- Per-source QoS tracking
- Best source selection
- Quality reporting API
### Phase 4: Testing (1 day)
- Simulated network conditions
- Latency injection
- Jitter simulation
- Quality degradation scenarios
## Files to Create/Modify
**New Files:**
- `Sources/VRMMetalKit/ARKit/QoSController.swift` (~400 lines)
- `Sources/VRMMetalKit/ARKit/ARMetadataSourceManager.swift` (~300 lines)
**Modify:**
- `Sources/VRMMetalKit/ARKit/ARKitFaceDriver.swift` - Integrate QoS controller
- `Sources/VRMMetalKit/ARKit/ARKitBodyDriver.swift` - Integrate QoS controller
- `CLAUDE.md` - Document QoS features
## Acceptance Criteria
- [ ] Latency tracking with statistics (avg, p50, p95, p99)
- [ ] Jitter measurement over rolling window
- [ ] Quality assessment based on metrics
- [ ] Adaptive staleness thresholds
- [ ] Frame interpolation for face blend shapes
- [ ] Frame interpolation for skeleton (position + rotation)
- [ ] Source manager with registration
- [ ] Quality reporting API
- [ ] Tests with simulated network conditions
- [ ] Documentation updated
## Performance Requirements
- QoS overhead: <50µs per frame
- Interpolation overhead: <100µs
- Memory: <5 KB per source
- No frame drops for good/excellent quality
## Priority
Medium - Nice to have for production, not blocking ArkavoCreator integration
## Dependencies
- Issue #25 (SLERP smoothing) for skeleton interpolation
## Related
- Part of ARKit Integration Phase 4 (QoS & Advanced Features)
- Mentioned in PR #24 as deferred work
- Documented in CLAUDE.md ARKit section
Contributor guide
Research direction
Start by reading Sources/VRMMetalKit/ARKit/ARKitFaceDriver.swift and ARKitBodyDriver.swift, then review issue #25 and the ARKit section of CLAUDE.md for existing smoothing and integration assumptions. Compare the proposed QoSController and ARMetadataSourceManager APIs with the current drivers, run the existing tests, and confirm that all listed acceptance criteria and performance requirements are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- mobile-dev, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100