Differential test harness: one reference + cross-target agreement for every op
- Dominant language
- C++
- Stars
- 5.8k
- Forks
- 471
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 81
Description
### Motivation
The per-op tests in `hwy/tests/` are thorough, but each one hand-writes its
own scalar reference and asserts only that *its* target matches *that*
reference. There's no direct target-vs-target check and no shared input
corpus, so two things slip through:
- a reference and an implementation can share the same bug (same author, same
mental model);
- a genuine cross-target divergence where target A and target B each pass
their own loose assertion on different inputs.
Contributors keep discovering target bugs the hard way, one CI round at a
time (RVV missing an op, an SVE sizeless-type issue, a NEON target-attribute
mismatch, subnormal handling differences). A differential harness on master
would surface those in Highway's own CI, decoupled from any PR.
### Sketch
One canonical scalar reference per op (`ref_f`, plain C++ over individual
lane values). A driver that, for each registered op:
1. builds input vectors from a corpus — structured boundary values (`0`,
`±1`, type min/max, `±inf`, NaN, subnormals, powers of two, values
straddling the op's own thresholds) + random + a small adversarial set;
2. runs the op on every enabled target and extracts lanes;
3. asserts `bits(target_out[i]) == bits(ref_f(in[i]))` for each lane, and
separately that **all targets produce identical bits** for the same
input;
4. sweeps full vs partial vectors and, on scalable targets, the valid lane
counts.
`test_util-inl.h` already has the lane-compare primitives; this adds the
registry, the corpus, and the cross-target loop on top.
### Design questions — these are yours to decide
I have a lean on each but would build it however you want:
1. **Rollout.** Framework + a starter set of ~15-20 ops, extended
incrementally? Or do you want it to aim for coverage from the start?
Which op family first (arithmetic? mask? swizzle?)?
2. **Op registry.** Generated from `quick_reference.md` (it's fairly
structured), a hand-maintained table, or a macro list? Is there any
machine-readable op metadata already that I've missed?
3. **Relationship to the existing ~65 test files.** A new supplementary
test binary that runs alongside them, or something the per-op tests
migrate onto over time?
4. **The canonical reference.** Consolidate the scattered scalar references
into one `ref_ops.h`, or keep them per-op and just have the harness call
in?
5. **Implementation-defined behaviour.** Some ops legitimately differ across
targets (float→int out-of-range, subnormal flush on some Arm configs,
`MulByFloorPow2` special cases). How would you want per-op tolerance /
"targets may differ here" declared — a field on the registry entry, a
trait, an allowlist?
6. **Input generation.** Op-aware generators (a declarative domain spec per
op, so `Log` gets `x > 0` and `ShiftRight` gets small counts), or a
generic structured+random corpus that's the same for everything?
7. **Scalable targets.** How far to push the lane-count sweep — rely on the
CI matrix's fixed VLs, or exercise multiple VLs within one run where the
toolchain allows?
8. **CI cost.** Always-on, sharded, or an opt-in slower job? Should it be
`AdjustedReps`-aware for debug/emulated builds?
9. **Failure output.** What format is actionable for you — input bits, the
disagreeing targets, their outputs, a minimal single-lane repro?
10. **Fuzzing.** Standalone, or also expose a libFuzzer entry point that
drives the same registry?
Would this be welcome, and where would you want it to start?
Contributor guide
Research direction
Start by reading the per-op tests in hwy/tests/ and the lane-compare primitives in test_util-inl.h, then review quick_reference.md for possible operation metadata. The issue currently defines a broad design space rather than a concrete starting task, so completion depends on agreeing the rollout, registry, reference, input, target, CI, and failure-reporting decisions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100