arkavo-org / arkavo-org/VRMMetalKit

Add multi-camera rendering helper methods for first-person view

Open
#70 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
6
Forks
2
Avg merge
18h 51m
Merged PRs (30d)
26

Description

## Issue Description

VRMMetalKit supports first-person flags but lacks helper methods for common multi-camera rendering patterns. This makes it harder to implement first-person/third-person view switching compared to UniVRM's layer-based system.

## Current State

✅ **Implemented:**
- First-person flag enum (auto, firstPersonOnly, thirdPersonOnly, both)
- Per-mesh first-person annotations
- Basic flag parsing from VRM files

❌ **Missing:**
- Helper methods for multi-camera setups
- Render filtering based on first-person flags
- Camera-specific rendering utilities
- Examples and documentation

## UniVRM Approach

UniVRM uses Unity's layer system:
```csharp
// UniVRM/Packages/VRM10/Runtime/Components/FirstPerson/
public class Vrm10FirstPersonLayerSettings {
public void SetupLayers(Vrm10Instance instance) {
// Assign meshes to layers based on flags
// Configure camera culling masks
}
}
```

## Proposed Solution

Since Metal doesn't have Unity's layer system, provide helper methods for manual filtering:

### 1. Render Filter API

```swift
// Sources/VRMMetalKit/Renderer/VRMRenderer+FirstPerson.swift

extension VRMRenderer {
/// Configure renderer for first-person camera
public func setFirstPersonMode(_ enabled: Bool) {
self.firstPersonMode = enabled
updateRenderFilter()
}

/// Get meshes visible in first-person view
public func getFirstPersonMeshes() -> [VRMMesh] {
return model?.meshes.filter { mesh in
mesh.firstPersonFlag == .firstPersonOnly ||
mesh.firstPersonFlag == .both ||
mesh.firstPersonFlag == .auto
} ?? []
}

/// Get meshes visible in third-person view
public func getThirdPersonMeshes() -> [VRMMesh] {
return model?.meshes.filter { mesh in
mesh.firstPersonFlag == .thirdPersonOnly ||
mesh.firstPersonFlag == .both ||
mesh.firstPersonFlag == .auto
} ?? []
}

/// Check if mesh should be rendered for current camera
public func shouldRenderMesh(_ mesh: VRMMesh, isFirstPerson: Bool) -> Bool {
switch mesh.firstPersonFlag {
case .auto, .both:
return true
case .firstPersonOnly:
return isFirstPerson
case .thirdPersonOnly:
return !isFirstPerson
}
}
}
```

### 2. Multi-Camera Setup Helper

```swift
// Sources/VRMMetalKit/Utilities/MultiCameraSetup.swift

public struct MultiCameraSetup {
public let firstPersonRenderer: VRMRenderer
public let thirdPersonRenderer: VRMRenderer

public init(device: MTLDevice, model: VRMModel) {
// Create two renderers with different filters
firstPersonRenderer = VRMRenderer(device: device)
firstPersonRenderer.setFirstPersonMode(true)
firstPersonRenderer.loadModel(model)

thirdPersonRenderer = VRMRenderer(device: device)
thirdPersonRenderer.setFirstPersonMode(false)
thirdPersonRenderer.loadModel(model)
}

public func render(
firstPersonView: MTKView,
thirdPersonView: MTKView,
commandBuffer: MTLCommandBuffer
) {
// Render first-person view
if let rpd = firstPersonView.currentRenderPassDescriptor {
firstPersonRenderer.draw(
in: firstPersonView,
commandBuffer: commandBuffer,
renderPassDescriptor: rpd
)
}

// Render third-person view
if let rpd = thirdPersonView.currentRenderPassDescriptor {
thirdPersonRenderer.draw(
in: thirdPersonView,
commandBuffer: commandBuffer,
renderPassDescriptor: rpd
)
}
}
}
```

### 3. Single-Renderer Approach

```swift
// Alternative: Use single renderer with dynamic filtering

extension VRMRenderer {
/// Render with first-person filtering
public func drawFirstPerson(
in view: MTKView,
commandBuffer: MTLCommandBuffer,
renderPassDescriptor: MTLRenderPassDescriptor
) {
let previousMode = self.firstPersonMode
self.firstPersonMode = true
defer { self.firstPersonMode = previousMode }

draw(in: view, commandBuffer: commandBuffer, renderPassDescriptor: renderPassDescriptor)
}

/// Render with third-person filtering
public func drawThirdPerson(
in view: MTKView,
commandBuffer: MTLCommandBuffer,
renderPassDescriptor: MTLRenderPassDescriptor
) {
let previousMode = self.firstPersonMode
self.firstPersonMode = false
defer { self.firstPersonMode = previousMode }

draw(in: view, commandBuffer: commandBuffer, renderPassDescriptor: renderPassDescriptor)
}
}
```

### 4. Head Mesh Handling

```swift
// Special handling for head mesh in first-person view

extension VRMRenderer {
/// Configure head mesh visibility
public func setHeadMeshVisible(_ visible: Bool) {
guard let model = self.model else { return }

// Find head mesh (typically marked as thirdPersonOnly)
for mesh in model.meshes {
if mesh.name?.lowercased().contains(head) == true ||
mesh.name?.lowercased().contains(face) == true {
mesh.isVisible = visible
}
}
}
}
```

## Usage Examples

### Example 1: VR Application with Dual Cameras

```swift
class VRViewController {
let multiCamera: MultiCameraSetup

func setupVR() async throws {
let model = try await VRMModel.load(from: avatarURL, device: device)
multiCamera = MultiCameraSetup(device: device, model: model)

// Configure cameras
multiCamera.firstPersonRenderer.viewMatrix = firstPersonCamera.viewMatrix
multiCamera.thirdPersonRenderer.viewMatrix = thirdPersonCamera.viewMatrix
}

func render() {
guard let commandBuffer = commandQueue.makeCommandBuffer() else { return }

multiCamera.render(
firstPersonView: leftEyeView,
thirdPersonView: mirrorView,
commandBuffer: commandBuffer
)

commandBuffer.commit()
}
}
```

### Example 2: Toggle First/Third Person

```swift
class GameViewController {
let renderer: VRMRenderer
var isFirstPerson = false

func toggleCamera() {
isFirstPerson.toggle()
renderer.setFirstPersonMode(isFirstPerson)

// Hide head in first-person
renderer.setHeadMeshVisible(!isFirstPerson)
}

func render() {
guard let commandBuffer = commandQueue.makeCommandBuffer(),
let rpd = metalView.currentRenderPassDescriptor else { return }

renderer.draw(in: metalView, commandBuffer: commandBuffer, renderPassDescriptor: rpd)
commandBuffer.commit()
}
}
```

### Example 3: Custom Filtering

```swift
// Advanced: Custom mesh filtering logic
extension VRMRenderer {
func renderWithCustomFilter(
filter: (VRMMesh) -> Bool,
in view: MTKView,
commandBuffer: MTLCommandBuffer,
renderPassDescriptor: MTLRenderPassDescriptor
) {
// Temporarily override mesh visibility
let originalVisibility = model?.meshes.map { /usr/bin/bash.isVisible } ?? []
defer {
model?.meshes.enumerated().forEach { index, mesh in
mesh.isVisible = originalVisibility[index]
}
}

model?.meshes.forEach { mesh in
mesh.isVisible = filter(mesh)
}

draw(in: view, commandBuffer: commandBuffer, renderPassDescriptor: renderPassDescriptor)
}
}
```

## Implementation Plan

### Phase 1: Core Helpers
- [ ] Add `setFirstPersonMode()` method
- [ ] Add `shouldRenderMesh()` filtering
- [ ] Add `getFirstPersonMeshes()` / `getThirdPersonMeshes()`
- [ ] Update renderer to respect first-person flags

### Phase 2: Multi-Camera Support
- [ ] Create `MultiCameraSetup` helper class
- [ ] Add `drawFirstPerson()` / `drawThirdPerson()` methods
- [ ] Add head mesh visibility control

### Phase 3: Documentation & Examples
- [ ] Add multi-camera example to Examples/
- [ ] Document first-person rendering patterns
- [ ] Add VR application example
- [ ] Create troubleshooting guide

### Phase 4: Testing
- [ ] Unit tests for mesh filtering
- [ ] Integration tests for multi-camera
- [ ] Visual tests with VR headset
- [ ] Performance benchmarks

## Acceptance Criteria

- [ ] Easy to set up first-person/third-person rendering
- [ ] Support for dual-camera VR applications
- [ ] Proper head mesh handling
- [ ] Clear documentation with examples
- [ ] No performance overhead when not using multi-camera
- [ ] Compatible with existing VRM first-person annotations

## Priority

**Low-Medium** - Nice to have for VR applications but not blocking core functionality

## Estimated Effort

- Phase 1: 3-4 days
- Phase 2: 3-4 days
- Phase 3: 2-3 days
- Phase 4: 2-3 days
- **Total:** 2 weeks

## Related Issues

- Related: #66 (spec compliance)
- Enhances: First-person view support

## References

- VRM 1.0 First Person Spec: https://github.com/vrm-c/vrm-specification/blob/master/specification/VRMC_vrm-1.0/firstPerson.md
- UniVRM First Person: https://github.com/vrm-c/UniVRM/tree/master/Assets/VRM10/Runtime/Components/FirstPerson

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing VRMRenderer and first-person flag implementation, then compare the proposed Sources/VRMMetalKit/Renderer/VRMRenderer+FirstPerson.swift and Sources/VRMMetalKit/Utilities/MultiCameraSetup.swift entry points. Use the issue's filtering, multi-camera, head-visibility, example, and testing phases to define completion; done requires documented first-/third-person rendering that respects existing annotations without affecting normal rendering.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
ar-vr-xr, computer-graphics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.