get2knowio / get2knowio/deacon
chore(parity): mine the reference CLI's own e2e test fixtures into declarative parity cases
- Dominant language
- Rust
- Stars
- 2
- Forks
- 0
- Avg merge
- 2h 36m
- Merged PRs (30d)
- 78
Description
The growth rule in `docs/PARITY.md` / CLAUDE.md says to "periodically mine the reference's own e2e fixtures and real-world configs for scenarios the suite does not have yet" — this issue makes that actionable for the richest single vein: the reference CLI's own test suite.
## The idea
`@devcontainers/cli` ships hundreds of e2e test configs (`src/test/**` in [devcontainers/cli](https://github.com/devcontainers/cli) — per-scenario `devcontainer.json`s, Dockerfiles, compose files, feature fixtures). Each one encodes a behavior upstream cared enough to pin. Running BOTH CLIs over them differentially asks, for free, "does deacon handle everything upstream's own tests exercise?" — coverage chosen by the reference's authors, not ours, which is exactly the blind-spot antidote.
## Method (data-only — this is the suite's whole design)
1. Harvest from the tag matching the oracle pin (**v0.87.0**, so fixtures and binary agree), not main.
2. Triage the harvest: drop configs exercising out-of-scope surfaces (feature authoring/test/publish), dedupe against existing `parity/fixtures/` coverage (many shapes are already covered — the win is the tail), group the rest by area.
3. For each survivor: vendor the fixture into `parity/fixtures/fx-upstream-*/` (MIT-licensed; keep upstream attribution in a fixture README line), author a `live-differential` case — a pure data edit, no new Rust.
4. Run the batch. Every divergence follows the standard triage from `docs/PARITY.md`: deacon wrong → issue + red case; reference deviates from spec → recorded conformant row; spec-silent deliberate → maintainer ruling before any tolerance.
5. Land in batches by area, each batch's SPEC_STATUS rows in the same commit. Expect the first batch to be the most informative and the cheapest.
## Guardrails
- Data and fixtures only; if a fixture needs a runner capability the model lacks, skip it and list it in the report — no machinery growth to chase a fixture.
- Environment-pinned fixtures (uid assumptions, huge images, arch-specific) get the same treatment the suite already documents: first run is a measurement, and `scripts/parity/prepull-fixture-images.sh` must discover anything image-hungry.
- Volume control: this is a periodic mining pass, not a bulk import — a batch that adds 500 cases nobody triaged is worse than 30 that were each read.
Second vein, same method, separate pass: real-world `devcontainer.json` files from popular public repos.
Contributor guide
Research direction
Start with docs/PARITY.md and CLAUDE.md, then inspect existing parity/fixtures/ coverage and the v0.87.0 src/test/** fixtures in devcontainers/cli. Triage out-of-scope and duplicate scenarios, vendor selected fixtures with attribution, and add live-differential cases. Done means each batch has been run, divergences triaged, and matching SPEC_STATUS rows recorded; use scripts/parity/prepull-fixture-images.sh for image-heavy cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- cli, devtools, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100