arkavo-org / arkavo-org/VRMMetalKit
ARKit: route Kalman face smoothing through MPS instead of CPU
- Dominant language
- Swift
- Stars
- 6
- Forks
- 2
- Avg merge
- 18h 51m
- Merged PRs (30d)
- 26
Description
## Source
Romero perf review, item 9.
## Problem
`SmoothingFilters.swift` has a `.kalman(processNoise:measurementNoise:)` case that runs the filter on the CPU per frame for each of 52 ARKit blend shapes. Cost is small per shape, but compounds with other CPU work in the ARKit-to-render path and adds latency.
## Proposed change
Dispatch the Kalman step through MPS (`MPSMatrixMultiplication` for the predict/update steps, or a small custom kernel) so it runs on GPU/ANE. Frees CPU and tightens the camera → ARFaceAnchor → frame-on-screen latency budget (Romero report §4).
## Affected
- `Sources/VRMMetalKit/ARKit/SmoothingFilters.swift`
- `Sources/VRMMetalKit/ARKit/ARKitFaceDriver.swift`
## Related
- #156 (measure on a known clip before/after)
Contributor guide
Research direction
Start by reading Sources/VRMMetalKit/ARKit/SmoothingFilters.swift and ARKitFaceDriver.swift, tracing the .kalman path through the per-frame blend-shape processing. Review #156 for the known-clip measurement approach, then compare the MPS dispatch against the CPU path. Done means the Kalman step uses MPS and the before/after CPU and latency measurements support the change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- mobile-dev, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100