arkavo-org / arkavo-org/VRMMetalKit

ARKit: route Kalman face smoothing through MPS instead of CPU

Open
#198 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
6
Forks
2
Avg merge
18h 51m
Merged PRs (30d)
26

Description

## Source
Romero perf review, item 9.

## Problem
`SmoothingFilters.swift` has a `.kalman(processNoise:measurementNoise:)` case that runs the filter on the CPU per frame for each of 52 ARKit blend shapes. Cost is small per shape, but compounds with other CPU work in the ARKit-to-render path and adds latency.

## Proposed change
Dispatch the Kalman step through MPS (`MPSMatrixMultiplication` for the predict/update steps, or a small custom kernel) so it runs on GPU/ANE. Frees CPU and tightens the camera → ARFaceAnchor → frame-on-screen latency budget (Romero report §4).

## Affected
- `Sources/VRMMetalKit/ARKit/SmoothingFilters.swift`
- `Sources/VRMMetalKit/ARKit/ARKitFaceDriver.swift`

## Related
- #156 (measure on a known clip before/after)

Contributor guide

Open the contributing guide

Research direction

Start by reading Sources/VRMMetalKit/ARKit/SmoothingFilters.swift and ARKitFaceDriver.swift, tracing the .kalman path through the per-frame blend-shape processing. Review #156 for the known-clip measurement approach, then compare the MPS dispatch against the CPU path. Done means the Kalman step uses MPS and the before/after CPU and latency measurements support the change.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
mobile-dev, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.