arkavo-org / arkavo-org/VRMMetalKit
ARKit: Weighted skeleton blending for multi-camera body tracking
- Dominant language
- Swift
- Stars
- 6
- Forks
- 2
- Avg merge
- 18h 51m
- Merged PRs (30d)
- 26
Description
## Description
Implement weighted blending of skeleton transforms from multiple cameras for improved body tracking accuracy and coverage. This allows combining body tracking data from different viewpoints (e.g., front camera + side camera).
## Problem Statement
Current multi-source body tracking uses priority strategies that select a single source:
- `.latestActive` - Use most recent
- `.primary(id, fallback)` - Use specific source
- `.highestConfidence` - Use best quality
But some scenarios benefit from blending:
- Front camera: Good upper body, poor lower body
- Side camera: Good lower body, poor upper body
- Combining both gives complete skeleton
The `.weighted([id: weight])` strategy exists but is a placeholder:
```swift
// ARKitBodyDriver.swift:272
case .weighted:
// TODO: Implement weighted blending of transforms
// For now, fallback to latestActive
return skeletons.values.max(by: { $0.timestamp < $1.timestamp })
```
## Challenges
Blending skeleton transforms is non-trivial:
1. **Quaternion blending requires SLERP** (issue #25)
2. **Per-joint confidence weighting** (not just per-source)
3. **Handle missing joints** in partial skeletons
4. **Coordinate system alignment** between cameras
5. **Performance** (needs to be <200µs for blending)
## Proposed Implementation
### 1. Per-Joint Weights
```swift
struct SkeletonBlendConfig {
/// Weight per source per joint
var weights: [String: [ARKitJoint: Float]] = [:]
/// Auto-weight based on joint confidence
var autoWeightByConfidence: Bool = true
/// Normalize weights to sum to 1.0
var normalizeWeights: Bool = true
}
```
### 2. Weighted Transform Blending
```swift
func blendSkeletons(
_ skeletons: [String: ARKitBodySkeleton],
config: SkeletonBlendConfig
) -> ARKitBodySkeleton {
var blendedJoints: [ARKitJoint: simd_float4x4] = [:]
for joint in ARKitJoint.allCases {
var transforms: [(transform: simd_float4x4, weight: Float)] = []
for (sourceID, skeleton) in skeletons {
guard let transform = skeleton.transform(for: joint) else { continue }
let weight = config.weights[sourceID]?[joint] ?? 0.0
if weight > 0 {
transforms.append((transform, weight))
}
}
if !transforms.isEmpty {
blendedJoints[joint] = blendTransforms(transforms)
}
}
return ARKitBodySkeleton(
timestamp: latestTimestamp(skeletons),
joints: blendedJoints,
isTracked: !blendedJoints.isEmpty,
confidence: averageConfidence(skeletons)
)
}
```
### 3. Transform Blending Algorithm
```swift
func blendTransforms(_ transforms: [(simd_float4x4, Float)]) -> simd_float4x4 {
// Decompose each transform
let decomposed = transforms.map { (transform, weight) in
let (pos, rot, scale) = decomposeTransform(transform)
return (pos, rot, scale, weight)
}
// Normalize weights
let totalWeight = decomposed.reduce(0) { $0 + $1.3 }
let normalized = decomposed.map { ($0.0, $0.1, $0.2, $0.3 / totalWeight) }
// Blend components
var blendedPos = SIMD3(0, 0, 0)
var blendedRot = simd_quatf(ix: 0, iy: 0, iz: 0, r: 1)
var blendedScale = SIMD3(0, 0, 0)
for (pos, rot, scale, weight) in normalized {
// Linear blend for position and scale
blendedPos += pos * weight
blendedScale += scale * weight
// SLERP for rotation (requires issue #25)
blendedRot = simd_slerp(blendedRot, rot, weight)
}
// Recompose to 4x4 matrix
return composeTransform(blendedPos, blendedRot, blendedScale)
}
```
### 4. Auto-Weighting Heuristics
Automatically determine weights based on:
**Joint-specific confidence:**
```swift
func autoWeight(for joint: ARKitJoint, in skeleton: ARKitBodySkeleton) -> Float {
// Use per-joint confidence if available
if let confidence = skeleton.jointConfidence(joint) {
return confidence
}
// Fallback to heuristics based on joint type
switch joint {
case .head, .neck, .chest:
// Upper body: prefer front camera
return skeleton.sourceType == .front ? 0.8 : 0.2
case .leftFoot, .rightFoot, .leftLowerLeg, .rightLowerLeg:
// Lower body: prefer side/back camera
return skeleton.sourceType == .side ? 0.8 : 0.2
default:
return 0.5
}
}
```
**Viewpoint coverage:**
```swift
enum CameraViewpoint {
case front, side, back, overhead
}
func optimalWeight(joint: ARKitJoint, viewpoint: CameraViewpoint) -> Float {
// Different viewpoints see different joints better
// Based on occlusion and angle
}
```
## Implementation Plan
### Phase 1: Core Blending (2 days)
- Transform decomposition/recomposition
- Linear blending for position/scale
- Quaternion SLERP blending (depends on #25)
- Normalize weights
### Phase 2: Auto-Weighting (1 day)
- Per-joint confidence extraction
- Viewpoint heuristics
- Confidence-based weighting
### Phase 3: Integration (1 day)
- Update ARKitBodyDriver.selectSource()
- Add SkeletonBlendConfig
- Performance optimization
- Memory pooling for blend operations
### Phase 4: Testing (1 day)
- Unit tests for blending math
- Multi-camera scenarios
- Performance validation
- Visual quality comparison
## Files to Modify
- `Sources/VRMMetalKit/ARKit/ARKitBodyDriver.swift`
- Implement weighted blending
- Add SkeletonBlendConfig
- Update selectSource() for .weighted case
- `Sources/VRMMetalKit/ARKit/ARKitTypes.swift`
- Add per-joint confidence (optional)
- Add camera viewpoint metadata
## Performance Requirements
- Blending overhead: <200µs for 50 joints
- Memory: <5 KB temporary allocations
- No heap allocations in hot path
- SIMD optimization for vector ops
## Acceptance Criteria
- [ ] Transform blending with SLERP implemented
- [ ] Auto-weighting heuristics working
- [ ] Config API for manual weights
- [ ] Performance meets requirements
- [ ] Visual quality improvement demonstrated
- [ ] Unit tests for blending math
- [ ] Integration tests with multi-camera
- [ ] Documentation updated
## Use Cases
**Desk scenario:**
- Front camera: Face + upper body (0.7 weight)
- Overhead camera: Hands + keyboard (0.3 weight)
- Blended result: Complete upper body tracking
**Full body:**
- Front camera: Upper body (0.6 weight)
- Side camera: Lower body + profile (0.4 weight)
- Blended result: Full skeleton with best coverage
## Priority
Low - Nice to have, not blocking customer integration
## Dependencies
- **Blocking:** Issue #25 (SLERP smoothing) for quaternion blending
- **Optional:** Per-joint confidence data from ARKit
## Related
- Part of ARKit Integration Phase 4 (QoS & Advanced Features)
- Mentioned in PR #24 as deferred work
- Requires SLERP implementation from issue #25
Contributor guide
Research direction
Start in Sources/VRMMetalKit/ARKit/ARKitBodyDriver.swift at the .weighted placeholder, then inspect ARKitTypes.swift and issue #25 for the required skeleton and SLERP support. Define the scope and existing APIs before implementing the listed blending, weighting, performance, and test requirements; done means the acceptance criteria pass, including unit and multi-camera integration tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- computer-vision, mobile-dev
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100