google / google/highway

Differential test harness: one reference + cross-target agreement for every op

Open
#3,353 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
5.8k
Forks
471
Avg merge
1d 6h
Merged PRs (30d)
81

Description

### Motivation

The per-op tests in `hwy/tests/` are thorough, but each one hand-writes its
own scalar reference and asserts only that *its* target matches *that*
reference. There's no direct target-vs-target check and no shared input
corpus, so two things slip through:

- a reference and an implementation can share the same bug (same author, same
mental model);
- a genuine cross-target divergence where target A and target B each pass
their own loose assertion on different inputs.

Contributors keep discovering target bugs the hard way, one CI round at a
time (RVV missing an op, an SVE sizeless-type issue, a NEON target-attribute
mismatch, subnormal handling differences). A differential harness on master
would surface those in Highway's own CI, decoupled from any PR.

### Sketch

One canonical scalar reference per op (`ref_f`, plain C++ over individual
lane values). A driver that, for each registered op:

1. builds input vectors from a corpus — structured boundary values (`0`,
`±1`, type min/max, `±inf`, NaN, subnormals, powers of two, values
straddling the op's own thresholds) + random + a small adversarial set;
2. runs the op on every enabled target and extracts lanes;
3. asserts `bits(target_out[i]) == bits(ref_f(in[i]))` for each lane, and
separately that **all targets produce identical bits** for the same
input;
4. sweeps full vs partial vectors and, on scalable targets, the valid lane
counts.

`test_util-inl.h` already has the lane-compare primitives; this adds the
registry, the corpus, and the cross-target loop on top.

### Design questions — these are yours to decide

I have a lean on each but would build it however you want:

1. **Rollout.** Framework + a starter set of ~15-20 ops, extended
incrementally? Or do you want it to aim for coverage from the start?
Which op family first (arithmetic? mask? swizzle?)?
2. **Op registry.** Generated from `quick_reference.md` (it's fairly
structured), a hand-maintained table, or a macro list? Is there any
machine-readable op metadata already that I've missed?
3. **Relationship to the existing ~65 test files.** A new supplementary
test binary that runs alongside them, or something the per-op tests
migrate onto over time?
4. **The canonical reference.** Consolidate the scattered scalar references
into one `ref_ops.h`, or keep them per-op and just have the harness call
in?
5. **Implementation-defined behaviour.** Some ops legitimately differ across
targets (float→int out-of-range, subnormal flush on some Arm configs,
`MulByFloorPow2` special cases). How would you want per-op tolerance /
"targets may differ here" declared — a field on the registry entry, a
trait, an allowlist?
6. **Input generation.** Op-aware generators (a declarative domain spec per
op, so `Log` gets `x > 0` and `ShiftRight` gets small counts), or a
generic structured+random corpus that's the same for everything?
7. **Scalable targets.** How far to push the lane-count sweep — rely on the
CI matrix's fixed VLs, or exercise multiple VLs within one run where the
toolchain allows?
8. **CI cost.** Always-on, sharded, or an opt-in slower job? Should it be
`AdjustedReps`-aware for debug/emulated builds?
9. **Failure output.** What format is actionable for you — input bits, the
disagreeing targets, their outputs, a minimal single-lane repro?
10. **Fuzzing.** Standalone, or also expose a libFuzzer entry point that
drives the same registry?

Would this be welcome, and where would you want it to start?

Contributor guide

Open the contributing guide

Research direction

Start by reading the per-op tests in hwy/tests/ and the lane-compare primitives in test_util-inl.h, then review quick_reference.md for possible operation metadata. The issue currently defines a broad design space rather than a concrete starting task, so completion depends on agreeing the rollout, registry, reference, input, target, CI, and failure-reporting decisions.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.