arkavo-org / arkavo-org/vrm-conformance

Methodology: rebase declared per-test SSIM threshold from 0.985 (self-diff) to cross-renderer-realistic values

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Problem

Every test plan emitted by `vrm-asset-generator` carries:
\`\`\`yaml
diff:
mode: ssim
threshold: 0.985
reference_renderer: vrm-metal-kit
\`\`\`

This threshold (0.985) was scoped for "this renderer producing byte-stable output across runs" — a self-diff stability target. It is **not achievable cross-renderer between independent implementations**.

Empirical evidence from the 4-renderer corpus run (`docs/findings.md` run 11):

| Pair | tests ≥ 0.985 | tests ≥ 0.95 | tests ≥ 0.85 |
|---|---|---|---|
| three-vrm ↔ UniVRM (closest pair, both spec-author-aligned) | 10/80 (12%) | 54/80 (68%) | 80/80 (100%, ex outline) |
| VMK ↔ UniVRM (consortium reference) | 0/80 (0%) | 7/80 (9%) | 54/80 (68%) |
| godot-vrm ↔ UniVRM | 0/80 (0%) | 7/80 (9%) | 73/80 (91%) |

**Zero** of 80 tests achieve 0.985 SSIM for the VMK↔reference pair. Even the closest cross-renderer pair in existence only crosses 0.985 on 12% of tests. Reporting "0/80 conformance" against this threshold is misleading — the threshold is the failure, not the renderer.

## Proposed change

Replace the uniform 0.985 with per-cluster operational thresholds, baked into the asset generator's test-plan emission:

| Test family | Proposed threshold | Rationale |
|---|---|---|
| MToon material (default, doubleSided, renderQueueOffset, shadingShift, giEqualization, rimLighting) | **0.85** | Bulk-band agreement; engine-level residual (silhouette AA, color pipeline) is the noise floor |
| MToon shadingToony | **0.85** | Same family; cluster currently at ~0.87 once VMK#205 closes |
| MToon outline (width ≤ 0.03, screen or world) | **0.85** | Bulk-band; outline + body comparison works at small widths |
| **MToon outline (width ≥ 0.05)** | **excluded** with rationale | Spec-correct flood; whole-frame SSIM measures silhouette-AA only — see companion issue on outline-test methodology |
| Spring-bone settle | **0.85** | Settled pose is mostly rest-pose for default chain mass; bulk-band agreement |
| Spring-bone swing | **0.85** | Once VMK#206 closes; needs cross-renderer animation alignment first |

Future tightening (post-1.0) can ratchet the MToon-material threshold to 0.90 once engine-level alignment work (MSAA pattern, color pipeline) is in scope.

## Implementation

1. Extend `vrm-test-plan::Diff` to accept either a uniform `threshold` (current) or a per-test override via a new field, and a `conformance_status` enum: `included | excluded`.
2. In `vrm-asset-generator`, set `threshold` per test_id at emission time. Outline tests at width ≥ 0.05 get `conformance_status: excluded` with a `excluded_reason` field pointing at the outline-methodology issue.
3. In `vrm-runner consensus-report`, surface "pass rate against declared threshold" as the primary headline; surface "excluded count" separately. Mean SSIM becomes a supporting stat.
4. Document the threshold rationale in `docs/methodology.md` so the launch-copy framing is reproducible.

## Acceptance

- `scripts/consensus-report.sh` reports "VMK ↔ UniVRM passes declared per-test threshold on N of 80 tests" as the primary line.
- `docs/findings.md` cites that N (not corpus-wide mean) as the conformance claim.
- A test plan with `conformance_status: excluded` is rendered + compared but not counted toward conformance pass-rate.

## Related

- [VMK#205](https://github.com/arkavo-org/VRMMetalKit/issues/205) — shadingToony cluster (upstream fix)
- [VMK#206](https://github.com/arkavo-org/VRMMetalKit/issues/206) — animate_root_transform no-op (upstream fix)
- Companion issue on outline-test methodology (whole-frame SSIM vs ring-band) filed separately.

Contributor guide

Open the contributing guide

Research direction

Start by tracing vrm-test-plan::Diff through vrm-asset-generator and the consensus-report flow, then inspect scripts/consensus-report.sh and docs/methodology.md. Verify how per-test thresholds and conformance_status are emitted and counted, and use the acceptance criteria to confirm excluded tests are rendered but omitted from pass rates and that the declared-threshold count is the primary report.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, shell
Domain
computer-graphics, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.