arkavo-org / arkavo-org/VRMMetalKit

Multi-avatar residency: GPU-native cache container + tiered residency (tens of VRMs concurrent)

Open
#337 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
6
Forks
2
Avg merge
18h 51m
Merged PRs (30d)
26

Description

## Problem
Target is tens of distinct VRMs rendered concurrently on Apple Silicon, M1/8GB
minimum. Current loader produces ~330MB resident per avatar (~12× the 26MB GLB)
from PNG decodes, full mip chains, and dense morph expansion. 30 avatars ≈ 10GB
— unworkable, and macOS has no jetsam: the failure mode is compressor/swap
degradation, not a crash.

## Architecture: two orthogonal axes
1. **Constant factor** — import-time transcode of VRM → page-aligned,
GPU-native cache container.
2. **Residency mechanism** — tiered residency over that container, driven by a
proactive budget governor.

Axis 2 without axis 1 inherits GLB's hostile layout (4-byte alignment,
interleaved float32, PNG textures); do not attempt mmap/bytesNoCopy residency
over the raw GLB.

## Axis 1: cache container (transcode-on-first-load, content-addressed)
- [ ] ASTC for **all** texture tiers incl. hero — toon-shaded VRM albedo is
ASTC's best case; tier axis collapses to mip floor only. No RGBA8 tier.
- [ ] Final vertex layout (half-float where precision allows), page-aligned
sections sized for `makeBuffer(bytesNoCopy:)`.
- [ ] **Sparse morph deltas** as (index, delta) pairs evaluated in a compute
scatter pass — VRM blendshapes are face-local; expect 10–20× vs dense.
*Gate:* measure actual VRoid export sparsity first (one afternoon). If
≥10×, this deletes the purgeable/re-expand morph subsystem entirely.
- [ ] Content-addressed chunks → runtime dedup: shared MTLBuffer/MTLTexture
across instances. VRoid avatars share base-mesh bytes heavily; 30
"distinct" avatars may be ~6 bodies + 30 face/hair sets.

## Axis 2: residency
- [ ] **Metal sparse textures** (Apple GPU family 7+) for mip-floor ladder —
tile map/unmap on a live texture, no texture-object swap, no pop.
Promotion = disk read of ASTC tiles, zero decode.
- [ ] `bytesNoCopy` geometry over the cache container (layout-correct
post-transcode). Note: Metal wires pages referenced by in-flight command
buffers — pager eviction applies only to unbound buffers; track
bindability explicitly.
- [ ] Far tier: instanced draws over deduped meshes; **impostors** (low-refresh
billboard re-render) as the rung below "statue" — a statue still pays
full skinned-draw cost.
- [ ] Tier ladder per avatar: hero (full mips, DQS, full spring-bone substeps)
→ mid (mip floor raised) → far (instanced/impostor, LBS, reduced
substeps, morphs cold).

## Governor
- [ ] **Proactive, budget-driven**: tier assignment from prominence score
(screen coverage × distance × is-speaking) against a budget derived from
`os_proc_available_memory()` at assignment time. DispatchSource pressure
events are a late-warning backstop only — they fire after degradation
has begun.
- [ ] Prominence accepts **externally declared hints** (host app may know
scene composition); falls back to measured prominence.
- [ ] Same code path 8GB M1 → 128GB M4 Max; ladder differs by budget only.

## Out of scope
- Cross-process / AR surfaces; USDZ path.
- Semantic LOD (mesh decimation) — mip/instancing/impostor only for v1.

## Open questions
- Sparse-texture tile size vs. ASTC block size interaction on family 7 vs 8.
- Impostor refresh policy: frame-count vs. pose-delta threshold.

## Acceptance
- 30 distinct VRoid avatars: ≤1GB steady-state on 8GB M1, no visible pop on
tier transitions, hero promotion latency ≤1 fram

Contributor guide

Open the contributing guide

Research direction

Start by measuring actual VRoid export morph sparsity, as the issue identifies this as a one-afternoon gate for choosing sparse morph deltas. Then trace the current loader and the Metal resource and residency entry points; the work is done when 30 distinct VRoid avatars stay within 1GB on an 8GB M1, tier transitions show no visible pop, and hero promotion is within one frame.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
computer-graphics, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.