arkavo-org / arkavo-org/VRMMetalKit
Add multi-camera rendering helper methods for first-person view
- Dominant language
- Swift
- Stars
- 6
- Forks
- 2
- Avg merge
- 18h 51m
- Merged PRs (30d)
- 26
Description
## Issue Description
VRMMetalKit supports first-person flags but lacks helper methods for common multi-camera rendering patterns. This makes it harder to implement first-person/third-person view switching compared to UniVRM's layer-based system.
## Current State
✅ **Implemented:**
- First-person flag enum (auto, firstPersonOnly, thirdPersonOnly, both)
- Per-mesh first-person annotations
- Basic flag parsing from VRM files
❌ **Missing:**
- Helper methods for multi-camera setups
- Render filtering based on first-person flags
- Camera-specific rendering utilities
- Examples and documentation
## UniVRM Approach
UniVRM uses Unity's layer system:
```csharp
// UniVRM/Packages/VRM10/Runtime/Components/FirstPerson/
public class Vrm10FirstPersonLayerSettings {
public void SetupLayers(Vrm10Instance instance) {
// Assign meshes to layers based on flags
// Configure camera culling masks
}
}
```
## Proposed Solution
Since Metal doesn't have Unity's layer system, provide helper methods for manual filtering:
### 1. Render Filter API
```swift
// Sources/VRMMetalKit/Renderer/VRMRenderer+FirstPerson.swift
extension VRMRenderer {
/// Configure renderer for first-person camera
public func setFirstPersonMode(_ enabled: Bool) {
self.firstPersonMode = enabled
updateRenderFilter()
}
/// Get meshes visible in first-person view
public func getFirstPersonMeshes() -> [VRMMesh] {
return model?.meshes.filter { mesh in
mesh.firstPersonFlag == .firstPersonOnly ||
mesh.firstPersonFlag == .both ||
mesh.firstPersonFlag == .auto
} ?? []
}
/// Get meshes visible in third-person view
public func getThirdPersonMeshes() -> [VRMMesh] {
return model?.meshes.filter { mesh in
mesh.firstPersonFlag == .thirdPersonOnly ||
mesh.firstPersonFlag == .both ||
mesh.firstPersonFlag == .auto
} ?? []
}
/// Check if mesh should be rendered for current camera
public func shouldRenderMesh(_ mesh: VRMMesh, isFirstPerson: Bool) -> Bool {
switch mesh.firstPersonFlag {
case .auto, .both:
return true
case .firstPersonOnly:
return isFirstPerson
case .thirdPersonOnly:
return !isFirstPerson
}
}
}
```
### 2. Multi-Camera Setup Helper
```swift
// Sources/VRMMetalKit/Utilities/MultiCameraSetup.swift
public struct MultiCameraSetup {
public let firstPersonRenderer: VRMRenderer
public let thirdPersonRenderer: VRMRenderer
public init(device: MTLDevice, model: VRMModel) {
// Create two renderers with different filters
firstPersonRenderer = VRMRenderer(device: device)
firstPersonRenderer.setFirstPersonMode(true)
firstPersonRenderer.loadModel(model)
thirdPersonRenderer = VRMRenderer(device: device)
thirdPersonRenderer.setFirstPersonMode(false)
thirdPersonRenderer.loadModel(model)
}
public func render(
firstPersonView: MTKView,
thirdPersonView: MTKView,
commandBuffer: MTLCommandBuffer
) {
// Render first-person view
if let rpd = firstPersonView.currentRenderPassDescriptor {
firstPersonRenderer.draw(
in: firstPersonView,
commandBuffer: commandBuffer,
renderPassDescriptor: rpd
)
}
// Render third-person view
if let rpd = thirdPersonView.currentRenderPassDescriptor {
thirdPersonRenderer.draw(
in: thirdPersonView,
commandBuffer: commandBuffer,
renderPassDescriptor: rpd
)
}
}
}
```
### 3. Single-Renderer Approach
```swift
// Alternative: Use single renderer with dynamic filtering
extension VRMRenderer {
/// Render with first-person filtering
public func drawFirstPerson(
in view: MTKView,
commandBuffer: MTLCommandBuffer,
renderPassDescriptor: MTLRenderPassDescriptor
) {
let previousMode = self.firstPersonMode
self.firstPersonMode = true
defer { self.firstPersonMode = previousMode }
draw(in: view, commandBuffer: commandBuffer, renderPassDescriptor: renderPassDescriptor)
}
/// Render with third-person filtering
public func drawThirdPerson(
in view: MTKView,
commandBuffer: MTLCommandBuffer,
renderPassDescriptor: MTLRenderPassDescriptor
) {
let previousMode = self.firstPersonMode
self.firstPersonMode = false
defer { self.firstPersonMode = previousMode }
draw(in: view, commandBuffer: commandBuffer, renderPassDescriptor: renderPassDescriptor)
}
}
```
### 4. Head Mesh Handling
```swift
// Special handling for head mesh in first-person view
extension VRMRenderer {
/// Configure head mesh visibility
public func setHeadMeshVisible(_ visible: Bool) {
guard let model = self.model else { return }
// Find head mesh (typically marked as thirdPersonOnly)
for mesh in model.meshes {
if mesh.name?.lowercased().contains(head) == true ||
mesh.name?.lowercased().contains(face) == true {
mesh.isVisible = visible
}
}
}
}
```
## Usage Examples
### Example 1: VR Application with Dual Cameras
```swift
class VRViewController {
let multiCamera: MultiCameraSetup
func setupVR() async throws {
let model = try await VRMModel.load(from: avatarURL, device: device)
multiCamera = MultiCameraSetup(device: device, model: model)
// Configure cameras
multiCamera.firstPersonRenderer.viewMatrix = firstPersonCamera.viewMatrix
multiCamera.thirdPersonRenderer.viewMatrix = thirdPersonCamera.viewMatrix
}
func render() {
guard let commandBuffer = commandQueue.makeCommandBuffer() else { return }
multiCamera.render(
firstPersonView: leftEyeView,
thirdPersonView: mirrorView,
commandBuffer: commandBuffer
)
commandBuffer.commit()
}
}
```
### Example 2: Toggle First/Third Person
```swift
class GameViewController {
let renderer: VRMRenderer
var isFirstPerson = false
func toggleCamera() {
isFirstPerson.toggle()
renderer.setFirstPersonMode(isFirstPerson)
// Hide head in first-person
renderer.setHeadMeshVisible(!isFirstPerson)
}
func render() {
guard let commandBuffer = commandQueue.makeCommandBuffer(),
let rpd = metalView.currentRenderPassDescriptor else { return }
renderer.draw(in: metalView, commandBuffer: commandBuffer, renderPassDescriptor: rpd)
commandBuffer.commit()
}
}
```
### Example 3: Custom Filtering
```swift
// Advanced: Custom mesh filtering logic
extension VRMRenderer {
func renderWithCustomFilter(
filter: (VRMMesh) -> Bool,
in view: MTKView,
commandBuffer: MTLCommandBuffer,
renderPassDescriptor: MTLRenderPassDescriptor
) {
// Temporarily override mesh visibility
let originalVisibility = model?.meshes.map { /usr/bin/bash.isVisible } ?? []
defer {
model?.meshes.enumerated().forEach { index, mesh in
mesh.isVisible = originalVisibility[index]
}
}
model?.meshes.forEach { mesh in
mesh.isVisible = filter(mesh)
}
draw(in: view, commandBuffer: commandBuffer, renderPassDescriptor: renderPassDescriptor)
}
}
```
## Implementation Plan
### Phase 1: Core Helpers
- [ ] Add `setFirstPersonMode()` method
- [ ] Add `shouldRenderMesh()` filtering
- [ ] Add `getFirstPersonMeshes()` / `getThirdPersonMeshes()`
- [ ] Update renderer to respect first-person flags
### Phase 2: Multi-Camera Support
- [ ] Create `MultiCameraSetup` helper class
- [ ] Add `drawFirstPerson()` / `drawThirdPerson()` methods
- [ ] Add head mesh visibility control
### Phase 3: Documentation & Examples
- [ ] Add multi-camera example to Examples/
- [ ] Document first-person rendering patterns
- [ ] Add VR application example
- [ ] Create troubleshooting guide
### Phase 4: Testing
- [ ] Unit tests for mesh filtering
- [ ] Integration tests for multi-camera
- [ ] Visual tests with VR headset
- [ ] Performance benchmarks
## Acceptance Criteria
- [ ] Easy to set up first-person/third-person rendering
- [ ] Support for dual-camera VR applications
- [ ] Proper head mesh handling
- [ ] Clear documentation with examples
- [ ] No performance overhead when not using multi-camera
- [ ] Compatible with existing VRM first-person annotations
## Priority
**Low-Medium** - Nice to have for VR applications but not blocking core functionality
## Estimated Effort
- Phase 1: 3-4 days
- Phase 2: 3-4 days
- Phase 3: 2-3 days
- Phase 4: 2-3 days
- **Total:** 2 weeks
## Related Issues
- Related: #66 (spec compliance)
- Enhances: First-person view support
## References
- VRM 1.0 First Person Spec: https://github.com/vrm-c/vrm-specification/blob/master/specification/VRMC_vrm-1.0/firstPerson.md
- UniVRM First Person: https://github.com/vrm-c/UniVRM/tree/master/Assets/VRM10/Runtime/Components/FirstPerson
Contributor guide
Research direction
Start by reading the existing VRMRenderer and first-person flag implementation, then compare the proposed Sources/VRMMetalKit/Renderer/VRMRenderer+FirstPerson.swift and Sources/VRMMetalKit/Utilities/MultiCameraSetup.swift entry points. Use the issue's filtering, multi-camera, head-visibility, example, and testing phases to define completion; done requires documented first-/third-person rendering that respects existing annotations without affecting normal rendering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- ar-vr-xr, computer-graphics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100