Golden fixtures in shared/test-data no longer match the JS reference (705/705)

Summary

The golden fixtures under shared/test-data/ no longer match what flint-js produces today. I hit this while working on a Waterfall fix (#79) and went looking for how far it reached, since it decides whether a JS-side change can be mirrored into flint-py.

Measured on dev @ c73d597, over the 705 fixture directories that have both input.json and expected.json:

Comparison	Identical	Differing
current flint-js == stored expected.json	0	705
current flint-py == current flint-js	0	705
current flint-py == stored expected.json	525	180

The third row is what pytest reports today (180 failures out of the 705 fixture tests), so it doubles as a calibration check on my comparator — it reproduces your number exactly.

Where the difference sits

Counted as "how many of the 705 fixtures differ at this path":

Fixtures	Path
705	config.legend
705	_transform
591	config.axisX/axisY.titleFontSize
524	_pivot
208 / 188	_options[].step / _options[].label
168 / 163	config.axisX/axisY.labelFontSize
148 / 133	config.view.continuousHeight / continuousWidth
133	config.facet.spacing

Every one is in the shared assembler / config layer. None is in a chart template. The per-chart Python ports look fine; what has moved is the layer underneath them.

What seems to have happened

packages/flint-py/tests/FULL_GALLERY_REPORT.md records 658/658 byte-for-byte parity with 0 mismatches, and its Methodology section names the tool that produced the fixtures:

The extractor at tests/frontend/unit/lib/flint-chart/flint_py_extract.test.ts walks GALLERY_TREE … Each case is written to flint-py/tests/fixtures/<slug>/ as input.json + expected.json

Two things about that, as the repo stands now:

1. The extractor is not in this repository. git log --all -- "*flint_py_extract*" returns nothing, and the path it is described at does not correspond to the current tree. It looks like it stayed behind when the project was restructured into a monorepo (53d50be, 2026-06-11).
2. tools/run_full_eval.py still points at the pre-rename location. packages/flint-py/tools/run_full_eval.py:23 has FIXTURES = ROOT / "tests" / "fixtures", but the corpus moved to shared/test-data/ in 1d1ee39 ("rename agents→agent-skills, test-fixtures→test-data") on the same day. That directory no longer exists, so the eval tool reports FIXTURE_MISSING.

Timeline from git log: the last commit that regenerated fixtures is f32e75f (2026-06-18, "port 7 Vega-Lite templates to JS parity + regen fixtures"); the corpus was last touched at all in dd09c9b (2026-06-26). Since then packages/flint-js/src has had 41 commits.

Why it stayed quiet

The half that broke is the generating side, not the checking side. pytest kept running and kept 525 fixtures green — against a corpus frozen in June. A stale golden file is the easiest kind to match, so the green signal held while the reference drifted away underneath it. Nothing in the suite fails when the corpus goes out of date, only when Python diverges from it.

One idea, if it is useful

meta.json already carries provenance (slug, generator, library, status) but not the JS revision the golden was recorded from. Adding something like sourceCommit and asserting that all fixtures share one value, and that it is not more than N commits behind HEAD, would make "the corpus is stale" a condition the suite can fail on — rather than something that has to be noticed.

Reproducing

1. In packages/flint-js, run a throwaway vitest that walks every shared/test-data/*/input.json and calls assembleVegaLite(fx.input), dumping the results to one JSON file. (705 inputs, 0 throw.)
2. In packages/flint-py, load each fixture's expected.json, compute assemble_vegalite(input), and compare all three with the canonicalisation from tests/test_fixtures.py (_canon: sorted keys, floats rounded to 9dp).
3. The calibration to check the comparator: Python == stored should come out at 705 − 180 = 525.

Happy to send the two scripts, or to open a PR that adds them under packages/flint-py/tools/ if that is somewhere you would want them.

I also have a Waterfall parity patch for flint-py ready (the totals inference, the connector layer, the value labels, and the title fix — its template output is byte-identical to current flint-js), but it is not much use until the corpus can be regenerated, since on its own it turns the 8 currently-green Waterfall fixtures red. Glad to send it whenever it is useful, or to leave it.

Start with shared/test-data, packages/flint-py/tools/run_full_eval.py, and tests/test_fixtures.py, then compare the fixture corpus with the current flint-js assembler as described. Check the missing extractor path and the fixture rename history before deciding whether regeneration, tooling changes, or provenance checks are in scope. Done means the intended fixture source and freshness checks are defined and the 705-case comparison has a reliable, documented result.