arkavo-org / arkavo-org/VRMMetalKit
Indirect Command Buffers (ICB) for multi-avatar crowd rendering
- Dominant language
- Swift
- Stars
- 6
- Forks
- 2
- Avg merge
- 18h 51m
- Merged PRs (30d)
- 26
Description
## Context
QA crowd benchmarks on M4 Mac mini (1024², animated + spring) show the avatar ceiling is entirely CPU-encode-bound:
| Avatars | render.median | FPS | CPU encode | GPU p95 |
|---------|--------------|-----|-----------|---------|
| 10 | 7.02ms | 142 | 4.08ms | 2.19ms |
| 20 | 12.26ms | 81.5 | 8.27ms | 3.09ms |
| 30 | 17.70ms | 57.0 | 12.44ms | 3.59ms |
| 50 | 28.54ms | 35.0 | 21.23ms | 4.74ms |
| 60 | 34.61ms | 28.8 | 26.08ms | 5.31ms |
**Ceiling**: ~28 avatars @ 60fps, ~58 @ 30fps on M4. M1 projection: ~20 @ 60fps.
GPU is 5× idle at 60 avatars. Single-threaded Metal command encoding (~0.41ms/avatar × 20 draws) is the bottleneck.
## Architecture Assessment
### What's ICB-Friendly (static per-material, doesn't change frame-to-frame)
- Pipeline state (PSO) — 11 variants, selected by alpha mode × skinning × features
- Vertex buffers (geometry) — vertex, index, normal, UV data
- Texture bindings — up to 9 fragment textures per material
- Sampler state — fixed, already hoisted outside the loop
- Depth stencil state — per face-category, cached
### What's Per-Frame Dynamic (must be patched, not baked)
- Uniforms (model/view/projection) — triple-buffered, 432 bytes
- MToon material uniforms — `setVertexBytes`/`setFragmentBytes` per draw (~400 bytes)
- Skinning joint matrices — GPU buffer, updated by compute pass
- Morphed position buffers — per-primitive, updated by compute pass
### Key Constraint: Inline Bytes vs Buffer-Bound
Metal ICBs cannot bake `setVertexBytes`/`setFragmentBytes` — they can only bake `setVertexBuffer`/`setFragmentBuffer` (buffer-bound). The current renderer uses `setVertexBytes` for MToon material uniforms on every draw. Converting to ICB requires moving material uniforms from inline bytes to a GPU buffer.
## Proposed Architecture: Hybrid ICB
### Phase 1: Material Uniform Buffer (prerequisite refactor)
Convert per-draw `setVertexBytes(&mtoonUniforms, index: 8)` / `setFragmentBytes` to a pre-filled `MTLBuffer` containing all material uniforms for all primitives, indexed by `drawIndex`. This is required before ICB and is independently beneficial (eliminates per-draw inline bytes copies).
**Changes:**
- New `MToonMaterialUniformBuffer`: one `MTLBuffer` sized for `N_primitives × MemoryLayout.stride`
- Per-frame: CPU writes all material uniforms into the buffer in primitive order, then binds once at `buffer(8)`
- Shader: no change — already reads from `buffer(8)`
### Phase 2: ICB Record + Replay
Once material uniforms are buffer-bound, the per-draw state that remains is:
- `setRenderPipelineState` — static per material
- `setVertexBuffer` (vertex, index, uniforms) — static per primitive
- `setFragmentTexture` × N — static per material
- `setDepthStencilState` / `setCullMode` / `setDepthBias` — static per face-category
- `drawIndexedPrimitives` — static geometry
All of this can be baked into an ICB at model load time.
**Per-frame dynamic updates via ICB patching:**
- `icb.computeEncoder` updates the uniforms buffer offset or model matrix in the ICB's argument buffer
- Only the uniform data changes; draw commands, PSOs, textures, geometry are all pre-baked
**Code flow:**
```
Model load:
1. Build render items (existing code)
2. Create MTLIndirectCommandBuffer(descriptor: ...)
3. icbEncoder = icb.newRenderCommandEncoder(descriptor: renderPassDescriptor)
4. For each render item:
- icbEncoder.setRenderPipelineState(specializedPSO)
- icbEncoder.setVertexBuffer(vertex, ...)
- icbEncoder.setFragmentTexture(baseColor, index: 0)
- ... all static bindings ...
- icbEncoder.drawIndexedPrimitives(...)
5. icbEncoder.endEncoding()
Per frame:
1. Update material uniform buffer (CPU memcpy)
2. Update model/view/projection uniform buffer (CPU memcpy)
3. Run spring bone compute (existing)
4. encoder.executeCommandsInBuffer(icb, range: 0..
Contributor guide
Research direction
Start with the existing render-item build and per-draw MToon uniform bindings described in Phase 1, then inspect the direct-encoding path and spring-bone compute pass. Treat the material uniform buffer as the first milestone; done means material data is buffer-bound without changing shader reads, with the direct fallback and Apple6+/mac2 capability gate preserved for later ICB record/replay.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- computer-graphics, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100