jlevy / jlevy/flowmark-rs

Shared parity corpus for flowmark + flowmark-rs (single source of truth)

Open
#60 0 comments 0 reactions 0 assignees View on GitHub
documentation enhancement
Dominant language
Rust
Stars
17
Forks
2
Avg merge
5h 10m
Merged PRs (30d)
7

Description

Tracking issue for the shared parity-corpus design — language-neutral test fixtures that drive both the Python (`jlevy/flowmark`) and Rust (`jlevy/flowmark-rs`) test suites from a single source of truth, so the two implementations are exercised against the *same* inputs with the *same* expected outputs and divergences are tracked explicitly rather than hidden in two parallel test trees.

Tracker: `fmr-bh2b`.

## Why

PR #59 surfaced three latent parity bugs (D19 image-ref inlining, the badge pattern, D20 ref-def label case) that traced to a single root cause: the upstream Python suite has near-zero coverage for reference-image syntactic forms, the Rust port mirrored Python's tests faithfully, and the gap propagated. A scratch differential run with all 655 CommonMark 0.31.2 spec examples found 69 Python↔Rust divergences (~10.5%) currently invisible to CI. The "port tests by hand" approach has reached its limit.

## Design (summary)

Full design doc on this branch: [`docs/project/specs/active/plan-2026-05-28-shared-parity-corpus.md`](../blob/claude/parity-followups-pr57-pr54/docs/project/specs/active/plan-2026-05-28-shared-parity-corpus.md). The spec is the authoritative design; the upstream issue (to be filed) is a suggestion that references it.

- **Corpus location:** `tests/parity_corpus/` in upstream Python — two-file pairs per case (`input.md` + `expected.md`) plus a TOML manifest. Initial seed: 655 CommonMark spec cases (auto-generated) + hand-curated flowmark families.
- **Generation mode:** default `flowmark_markdown()` settings. The corpus locks the most-used surface; alternate modes (semantic, plaintext, widths, list-spacing) stay with the existing targeted tests.
- **Python runner** (~20 LOC) — parametrized pytest, picked up by `make test`. All cases pass by construction.
- **Rust runner** (~120 LOC) — consumes the corpus through the existing `repos/flowmark` submodule; asserts `actual == expected` for non-baselined cases AND `actual != expected` for baselined cases. The baseline (`tests/parity_corpus_known_divergences.txt`) is monotone-decreasing.
- **No Python runtime dependency** in the Rust test job. Expecteds are pre-generated and committed.

## Process

### Phase 1 — upstream Python (file as one issue on `jlevy/flowmark`)

1. **Generate** all expecteds by running default-settings `flowmark_markdown()` across the corpus inputs.
2. **Sanity-check** the generated expecteds for obvious bugs in Python flowmark (dropped content, mangled escapes, lost indentation). Look hardest in tricky spec sections (backslash escapes, blockquoted fenced code, entity refs, indented code, tabs).
3. **Fix any bugs found**, regenerate the affected expecteds. The fixed output is the new locked baseline.
4. **Commit and lock in**, then **cut a minor patch release** (e.g. `0.7.1`) so the Rust side has a tagged commit to pin to.

### Phase 2 — Rust (this repo, after Phase 1 release lands)

Follows the standard [`rust-porting-playbook`](https://github.com/jlevy/rust-porting-playbook) workflow for picking up a new upstream release:

1. Bump `repos/flowmark` submodule to the Phase 1 release tag.
2. Differential corpus sweep between the new Python release and current Rust HEAD per the playbook's cross-validation step.
3. Port the runner to idiomatic Rust (not a line-for-line translation of the Python runner — the contract is the corpus, not the runner code).
4. Seed `tests/parity_corpus_known_divergences.txt` from the sweep output. Empty rationale lines not allowed.
5. Update `docs/parity-coverage-matrix.md` and `docs/port-status.md` to point at the new gate.

### Phase 3 — baseline triage (ongoing)

Cluster PRs that shrink the baseline. For each: minimal repro → root cause → fix in Rust OR document as permanent library-level divergence OR file upstream as a Python bug. Target residual baseline ≤20 entries within two quarterly syncs.

## Rollout

1. File one issue on `jlevy/flowmark` summarizing the corpus + sanity-check + release process, linking back to this spec doc as the authoritative design.
2. Upstream agent executes Phase 1, cuts the release.
3. Rust side picks up Phase 2 via the standard porting-playbook flow.
4. Phase 3 ongoing.

## Related

- **PR #59** — the parity-bugs PR whose review surfaced the structural gap, and the branch this spec doc currently lives on.
- **`docs/parity-coverage-matrix.md`** (added in #59) — the curated targeted matrix; will complement (not be replaced by) the corpus.
- **`tests/test_syntactic_surface.rs`** (added in #59) — the in-repo backstop for the families that will move into `tests/parity_corpus/cases/flowmark/`.
- **`jlevy/rust-porting-playbook#14`** (merged) — playbook now documents the pre-emptive syntactic-surface enumeration that this design operationalizes; Phase 2 here follows the playbook's standard sync workflow.
- **CommonMark spec 0.31.2** — https://spec.commonmark.org/0.31.2/

## Acceptance

- [ ] Upstream issue filed on `jlevy/flowmark`
- [ ] Phase 1 lands upstream and a minor release ships (with any sanity-check bug fixes)
- [ ] Phase 2 PR merged here (Rust runner + baseline) via the standard porting-playbook workflow
- [ ] `docs/parity-coverage-matrix.md` cross-references the live corpus runner
- [ ] Phase 3 baseline triage underway (PRs landing in clusters)

cc @jlevy

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.