arkavo-org / arkavo-org/VRMMetalKit

Add mutation testing to validate test-suite effectiveness

Open
#282 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
6
Forks
2
Avg merge
18h 51m
Merged PRs (30d)
26

Description

## Problem

Our test suite passes consistently and coverage looks good, but coverage measures *which lines run*, not *which lines have assertions that would catch a regression*. Existing experience (e.g. the per-PR `AvatarSample_A.png` visual diff regularly catches issues the automated suite doesn't) confirms there are gaps the line/branch metrics don't surface.

Mutation testing addresses this directly: it programmatically introduces small syntactic changes (mutants) into source-under-test, then runs the suite against each mutant. A useful test detects the change. If all tests pass on a mutated version, the mutation "survived" and exposes a real gap.

## How it works

Parse the source at AST level → apply a mutation operator at each eligible site → build → run tests → record kill/survive → revert → move on. A mutant is *killed* if any test fails (including timeouts and crashes) and *survives* if the suite passes unchanged. The mutation score is `killed / (total - equivalent)`.

Common operator families:

- Arithmetic replacement (`+ ↔ -`, `* ↔ /`)
- Relational/boundary shifts (`< ↔ <=`, `== ↔ !=`)
- Logical connector swaps (`&& ↔ ||`)
- Constant perturbation (`0 → 1`, `true → false`)
- Return-value replacement (`return` default / zero / `nil`)
- Statement deletion, call removal
- Aggressive sets: loop-bound mutations, exception swallowing

Tools differ mainly in operator set, AST coverage, and how aggressively they prune unreachable or trivially equivalent mutants.

## Handling survivors

Each surviving mutant falls into one of three buckets:

- **Real test gap** — add an assertion that distinguishes the mutant.
- **Equivalent mutant** — the change produces semantically identical behavior on all reachable inputs (e.g., mutating an unreachable branch, or `x * 1 → x * 1.0` where the cast is a no-op). Equivalence is undecidable in general; classify manually or with heuristics.
- **Dead code** — the operator hit a path that should be deleted rather than tested.

## Practical execution

Mutation testing is embarrassingly parallel but expensive — naive runs are O(mutants × test-suite-time). Mitigations:

- Compile-time mutant insertion to avoid full rebuilds
- Impact-localized test selection per mutant (via coverage-derived selection)
- Early-exit on first failure
- Parallel sandboxed workers

## Scope for VRMMetalKit

Initial proposal: integrate a Swift mutation testing tool (e.g. [`muter`](https://github.com/muter-mutation-testing/muter)) against one well-bounded subsystem first to validate the approach before expanding. Candidates ranked by leverage:

1. `Sources/VRMMetalKit/Animation/` — retargeting math is high-leverage and bugs are subtle.
2. `Sources/VRMMetalKit/Renderer/` — non-shader Swift (cache keying, descriptor building, strict-mode validation).
3. `Sources/GLTFCore/` — pure value-type helpers; easiest first target to prove out the loop.

Shader (`.metal`) sources are out of scope for an initial pass — operators don't generalize to MSL and the mutation loop would need GPU compilation per mutant.

## Acceptance criteria

- Reproducible `make mutation-test` target (or equivalent) that runs mutation testing against one chosen subsystem and emits a report.
- Documented baseline mutation score for that subsystem.
- A follow-up pass on the top N surviving mutants — kill, classify as equivalent, or remove dead code.
- Local-only execution first; CI integration deferred until cost is understood.

## Out of scope

- Mutation testing on `.metal` shaders.
- CI gate enforcement (a separate decision once baseline numbers exist).
- Whole-codebase coverage in one go.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the candidate subsystems in Sources/VRMMetalKit/Animation/, Sources/VRMMetalKit/Renderer/, and Sources/GLTFCore/, then evaluate muter for Swift support. Define a local make mutation-test target for one chosen subsystem, run it to establish a baseline report, and inspect the top surviving mutants. Done means the target is reproducible, the baseline is documented, and survivors are killed, classified as equivalent, or identified as dead code.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
build-system, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.