arkavo-org / arkavo-org/vrm-conformance
Methodology: rebase declared per-test SSIM threshold from 0.985 (self-diff) to cross-renderer-realistic values
- Dominant language
- Rust
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Every test plan emitted by `vrm-asset-generator` carries:
\`\`\`yaml
diff:
mode: ssim
threshold: 0.985
reference_renderer: vrm-metal-kit
\`\`\`
This threshold (0.985) was scoped for "this renderer producing byte-stable output across runs" — a self-diff stability target. It is **not achievable cross-renderer between independent implementations**.
Empirical evidence from the 4-renderer corpus run (`docs/findings.md` run 11):
| Pair | tests ≥ 0.985 | tests ≥ 0.95 | tests ≥ 0.85 |
|---|---|---|---|
| three-vrm ↔ UniVRM (closest pair, both spec-author-aligned) | 10/80 (12%) | 54/80 (68%) | 80/80 (100%, ex outline) |
| VMK ↔ UniVRM (consortium reference) | 0/80 (0%) | 7/80 (9%) | 54/80 (68%) |
| godot-vrm ↔ UniVRM | 0/80 (0%) | 7/80 (9%) | 73/80 (91%) |
**Zero** of 80 tests achieve 0.985 SSIM for the VMK↔reference pair. Even the closest cross-renderer pair in existence only crosses 0.985 on 12% of tests. Reporting "0/80 conformance" against this threshold is misleading — the threshold is the failure, not the renderer.
## Proposed change
Replace the uniform 0.985 with per-cluster operational thresholds, baked into the asset generator's test-plan emission:
| Test family | Proposed threshold | Rationale |
|---|---|---|
| MToon material (default, doubleSided, renderQueueOffset, shadingShift, giEqualization, rimLighting) | **0.85** | Bulk-band agreement; engine-level residual (silhouette AA, color pipeline) is the noise floor |
| MToon shadingToony | **0.85** | Same family; cluster currently at ~0.87 once VMK#205 closes |
| MToon outline (width ≤ 0.03, screen or world) | **0.85** | Bulk-band; outline + body comparison works at small widths |
| **MToon outline (width ≥ 0.05)** | **excluded** with rationale | Spec-correct flood; whole-frame SSIM measures silhouette-AA only — see companion issue on outline-test methodology |
| Spring-bone settle | **0.85** | Settled pose is mostly rest-pose for default chain mass; bulk-band agreement |
| Spring-bone swing | **0.85** | Once VMK#206 closes; needs cross-renderer animation alignment first |
Future tightening (post-1.0) can ratchet the MToon-material threshold to 0.90 once engine-level alignment work (MSAA pattern, color pipeline) is in scope.
## Implementation
1. Extend `vrm-test-plan::Diff` to accept either a uniform `threshold` (current) or a per-test override via a new field, and a `conformance_status` enum: `included | excluded`.
2. In `vrm-asset-generator`, set `threshold` per test_id at emission time. Outline tests at width ≥ 0.05 get `conformance_status: excluded` with a `excluded_reason` field pointing at the outline-methodology issue.
3. In `vrm-runner consensus-report`, surface "pass rate against declared threshold" as the primary headline; surface "excluded count" separately. Mean SSIM becomes a supporting stat.
4. Document the threshold rationale in `docs/methodology.md` so the launch-copy framing is reproducible.
## Acceptance
- `scripts/consensus-report.sh` reports "VMK ↔ UniVRM passes declared per-test threshold on N of 80 tests" as the primary line.
- `docs/findings.md` cites that N (not corpus-wide mean) as the conformance claim.
- A test plan with `conformance_status: excluded` is rendered + compared but not counted toward conformance pass-rate.
## Related
- [VMK#205](https://github.com/arkavo-org/VRMMetalKit/issues/205) — shadingToony cluster (upstream fix)
- [VMK#206](https://github.com/arkavo-org/VRMMetalKit/issues/206) — animate_root_transform no-op (upstream fix)
- Companion issue on outline-test methodology (whole-frame SSIM vs ring-band) filed separately.
Contributor guide
Research direction
Start by tracing vrm-test-plan::Diff through vrm-asset-generator and the consensus-report flow, then inspect scripts/consensus-report.sh and docs/methodology.md. Verify how per-test thresholds and conformance_status are emitted and counted, and use the acceptance criteria to confirm excluded tests are rendered but omitted from pass rates and that the declared-threshold count is the primary report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, shell
- Domain
- computer-graphics, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100