arkavo-org / arkavo-org/VRMMetalKit

ARKit: Weighted skeleton blending for multi-camera body tracking

Open
#29 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
6
Forks
2
Avg merge
18h 51m
Merged PRs (30d)
26

Description

## Description

Implement weighted blending of skeleton transforms from multiple cameras for improved body tracking accuracy and coverage. This allows combining body tracking data from different viewpoints (e.g., front camera + side camera).

## Problem Statement

Current multi-source body tracking uses priority strategies that select a single source:
- `.latestActive` - Use most recent
- `.primary(id, fallback)` - Use specific source
- `.highestConfidence` - Use best quality

But some scenarios benefit from blending:
- Front camera: Good upper body, poor lower body
- Side camera: Good lower body, poor upper body
- Combining both gives complete skeleton

The `.weighted([id: weight])` strategy exists but is a placeholder:
```swift
// ARKitBodyDriver.swift:272
case .weighted:
// TODO: Implement weighted blending of transforms
// For now, fallback to latestActive
return skeletons.values.max(by: { $0.timestamp < $1.timestamp })
```

## Challenges

Blending skeleton transforms is non-trivial:

1. **Quaternion blending requires SLERP** (issue #25)
2. **Per-joint confidence weighting** (not just per-source)
3. **Handle missing joints** in partial skeletons
4. **Coordinate system alignment** between cameras
5. **Performance** (needs to be <200µs for blending)

## Proposed Implementation

### 1. Per-Joint Weights

```swift
struct SkeletonBlendConfig {
/// Weight per source per joint
var weights: [String: [ARKitJoint: Float]] = [:]

/// Auto-weight based on joint confidence
var autoWeightByConfidence: Bool = true

/// Normalize weights to sum to 1.0
var normalizeWeights: Bool = true
}
```

### 2. Weighted Transform Blending

```swift
func blendSkeletons(
_ skeletons: [String: ARKitBodySkeleton],
config: SkeletonBlendConfig
) -> ARKitBodySkeleton {
var blendedJoints: [ARKitJoint: simd_float4x4] = [:]

for joint in ARKitJoint.allCases {
var transforms: [(transform: simd_float4x4, weight: Float)] = []

for (sourceID, skeleton) in skeletons {
guard let transform = skeleton.transform(for: joint) else { continue }
let weight = config.weights[sourceID]?[joint] ?? 0.0
if weight > 0 {
transforms.append((transform, weight))
}
}

if !transforms.isEmpty {
blendedJoints[joint] = blendTransforms(transforms)
}
}

return ARKitBodySkeleton(
timestamp: latestTimestamp(skeletons),
joints: blendedJoints,
isTracked: !blendedJoints.isEmpty,
confidence: averageConfidence(skeletons)
)
}
```

### 3. Transform Blending Algorithm

```swift
func blendTransforms(_ transforms: [(simd_float4x4, Float)]) -> simd_float4x4 {
// Decompose each transform
let decomposed = transforms.map { (transform, weight) in
let (pos, rot, scale) = decomposeTransform(transform)
return (pos, rot, scale, weight)
}

// Normalize weights
let totalWeight = decomposed.reduce(0) { $0 + $1.3 }
let normalized = decomposed.map { ($0.0, $0.1, $0.2, $0.3 / totalWeight) }

// Blend components
var blendedPos = SIMD3(0, 0, 0)
var blendedRot = simd_quatf(ix: 0, iy: 0, iz: 0, r: 1)
var blendedScale = SIMD3(0, 0, 0)

for (pos, rot, scale, weight) in normalized {
// Linear blend for position and scale
blendedPos += pos * weight
blendedScale += scale * weight

// SLERP for rotation (requires issue #25)
blendedRot = simd_slerp(blendedRot, rot, weight)
}

// Recompose to 4x4 matrix
return composeTransform(blendedPos, blendedRot, blendedScale)
}
```

### 4. Auto-Weighting Heuristics

Automatically determine weights based on:

**Joint-specific confidence:**
```swift
func autoWeight(for joint: ARKitJoint, in skeleton: ARKitBodySkeleton) -> Float {
// Use per-joint confidence if available
if let confidence = skeleton.jointConfidence(joint) {
return confidence
}

// Fallback to heuristics based on joint type
switch joint {
case .head, .neck, .chest:
// Upper body: prefer front camera
return skeleton.sourceType == .front ? 0.8 : 0.2
case .leftFoot, .rightFoot, .leftLowerLeg, .rightLowerLeg:
// Lower body: prefer side/back camera
return skeleton.sourceType == .side ? 0.8 : 0.2
default:
return 0.5
}
}
```

**Viewpoint coverage:**
```swift
enum CameraViewpoint {
case front, side, back, overhead
}

func optimalWeight(joint: ARKitJoint, viewpoint: CameraViewpoint) -> Float {
// Different viewpoints see different joints better
// Based on occlusion and angle
}
```

## Implementation Plan

### Phase 1: Core Blending (2 days)
- Transform decomposition/recomposition
- Linear blending for position/scale
- Quaternion SLERP blending (depends on #25)
- Normalize weights

### Phase 2: Auto-Weighting (1 day)
- Per-joint confidence extraction
- Viewpoint heuristics
- Confidence-based weighting

### Phase 3: Integration (1 day)
- Update ARKitBodyDriver.selectSource()
- Add SkeletonBlendConfig
- Performance optimization
- Memory pooling for blend operations

### Phase 4: Testing (1 day)
- Unit tests for blending math
- Multi-camera scenarios
- Performance validation
- Visual quality comparison

## Files to Modify

- `Sources/VRMMetalKit/ARKit/ARKitBodyDriver.swift`
- Implement weighted blending
- Add SkeletonBlendConfig
- Update selectSource() for .weighted case

- `Sources/VRMMetalKit/ARKit/ARKitTypes.swift`
- Add per-joint confidence (optional)
- Add camera viewpoint metadata

## Performance Requirements

- Blending overhead: <200µs for 50 joints
- Memory: <5 KB temporary allocations
- No heap allocations in hot path
- SIMD optimization for vector ops

## Acceptance Criteria

- [ ] Transform blending with SLERP implemented
- [ ] Auto-weighting heuristics working
- [ ] Config API for manual weights
- [ ] Performance meets requirements
- [ ] Visual quality improvement demonstrated
- [ ] Unit tests for blending math
- [ ] Integration tests with multi-camera
- [ ] Documentation updated

## Use Cases

**Desk scenario:**
- Front camera: Face + upper body (0.7 weight)
- Overhead camera: Hands + keyboard (0.3 weight)
- Blended result: Complete upper body tracking

**Full body:**
- Front camera: Upper body (0.6 weight)
- Side camera: Lower body + profile (0.4 weight)
- Blended result: Full skeleton with best coverage

## Priority

Low - Nice to have, not blocking customer integration

## Dependencies

- **Blocking:** Issue #25 (SLERP smoothing) for quaternion blending
- **Optional:** Per-joint confidence data from ARKit

## Related

- Part of ARKit Integration Phase 4 (QoS & Advanced Features)
- Mentioned in PR #24 as deferred work
- Requires SLERP implementation from issue #25

Contributor guide

Open the contributing guide

Research direction

Start in Sources/VRMMetalKit/ARKit/ARKitBodyDriver.swift at the .weighted placeholder, then inspect ARKitTypes.swift and issue #25 for the required skeleton and SLERP support. Define the scope and existing APIs before implementing the listed blending, weighting, performance, and test requirements; done means the acceptance criteria pass, including unit and multi-camera integration tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
computer-vision, mobile-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.