arkavo-org / arkavo-org/VRMMetalKit
Add mutation testing to validate test-suite effectiveness
- Dominant language
- Swift
- Stars
- 6
- Forks
- 2
- Avg merge
- 18h 51m
- Merged PRs (30d)
- 26
Description
## Problem
Our test suite passes consistently and coverage looks good, but coverage measures *which lines run*, not *which lines have assertions that would catch a regression*. Existing experience (e.g. the per-PR `AvatarSample_A.png` visual diff regularly catches issues the automated suite doesn't) confirms there are gaps the line/branch metrics don't surface.
Mutation testing addresses this directly: it programmatically introduces small syntactic changes (mutants) into source-under-test, then runs the suite against each mutant. A useful test detects the change. If all tests pass on a mutated version, the mutation "survived" and exposes a real gap.
## How it works
Parse the source at AST level → apply a mutation operator at each eligible site → build → run tests → record kill/survive → revert → move on. A mutant is *killed* if any test fails (including timeouts and crashes) and *survives* if the suite passes unchanged. The mutation score is `killed / (total - equivalent)`.
Common operator families:
- Arithmetic replacement (`+ ↔ -`, `* ↔ /`)
- Relational/boundary shifts (`< ↔ <=`, `== ↔ !=`)
- Logical connector swaps (`&& ↔ ||`)
- Constant perturbation (`0 → 1`, `true → false`)
- Return-value replacement (`return` default / zero / `nil`)
- Statement deletion, call removal
- Aggressive sets: loop-bound mutations, exception swallowing
Tools differ mainly in operator set, AST coverage, and how aggressively they prune unreachable or trivially equivalent mutants.
## Handling survivors
Each surviving mutant falls into one of three buckets:
- **Real test gap** — add an assertion that distinguishes the mutant.
- **Equivalent mutant** — the change produces semantically identical behavior on all reachable inputs (e.g., mutating an unreachable branch, or `x * 1 → x * 1.0` where the cast is a no-op). Equivalence is undecidable in general; classify manually or with heuristics.
- **Dead code** — the operator hit a path that should be deleted rather than tested.
## Practical execution
Mutation testing is embarrassingly parallel but expensive — naive runs are O(mutants × test-suite-time). Mitigations:
- Compile-time mutant insertion to avoid full rebuilds
- Impact-localized test selection per mutant (via coverage-derived selection)
- Early-exit on first failure
- Parallel sandboxed workers
## Scope for VRMMetalKit
Initial proposal: integrate a Swift mutation testing tool (e.g. [`muter`](https://github.com/muter-mutation-testing/muter)) against one well-bounded subsystem first to validate the approach before expanding. Candidates ranked by leverage:
1. `Sources/VRMMetalKit/Animation/` — retargeting math is high-leverage and bugs are subtle.
2. `Sources/VRMMetalKit/Renderer/` — non-shader Swift (cache keying, descriptor building, strict-mode validation).
3. `Sources/GLTFCore/` — pure value-type helpers; easiest first target to prove out the loop.
Shader (`.metal`) sources are out of scope for an initial pass — operators don't generalize to MSL and the mutation loop would need GPU compilation per mutant.
## Acceptance criteria
- Reproducible `make mutation-test` target (or equivalent) that runs mutation testing against one chosen subsystem and emits a report.
- Documented baseline mutation score for that subsystem.
- A follow-up pass on the top N surviving mutants — kill, classify as equivalent, or remove dead code.
- Local-only execution first; CI integration deferred until cost is understood.
## Out of scope
- Mutation testing on `.metal` shaders.
- CI gate enforcement (a separate decision once baseline numbers exist).
- Whole-codebase coverage in one go.
Contributor guide
Research direction
Start by comparing the candidate subsystems in Sources/VRMMetalKit/Animation/, Sources/VRMMetalKit/Renderer/, and Sources/GLTFCore/, then evaluate muter for Swift support. Define a local make mutation-test target for one chosen subsystem, run it to establish a baseline report, and inspect the top surviving mutants. Done means the target is reproducible, the baseline is documented, and survivors are killed, classified as equivalent, or identified as dead code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- build-system, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100