standards/CLAUDE.md: reproducing reported bugs — suspect the harness, and prove the test fails on base
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 7
Description
Problem
A repro harness that diverges from production on a single detail produces a confident false negative — the bug looks refuted, and the report gets closed or re-litigated from a wrong premise. This has now produced a wrong answer four times across recent sessions in AudioClassifier:
| Case | The one detail that diverged |
|---|---|
_headers vs. the wire |
Local server didn't send what production sends |
| FP32 vs. FP16 | Harness dtype differed from the deployed model |
Stubbed AbortSignal |
Stub omitted behavior the real API has |
#196 (net::ERR_FAILED offline) |
Test server sent a bare 308, production sends 308 + cache-control: no-cache |
The #196 case is the clearest. The claim was that the installed PWA can't launch offline because cache.addAll follows a 308 and precaches a redirected: true response, which respondWith rejects for a navigation. The harness sent a bare 308 and the app launched fine — reported as not reproducible. That was wrong: a bare 308 is a permanent redirect and therefore heuristically cacheable, so Chromium cached it and replayed /index.html → / offline without ever consulting the service worker, landing on a clean cache entry. Production sends cache-control: no-cache, which defeats the replay, so the navigation reaches the SW and hits the poisoned entry. Adding that one header made the test fail on main with the exact reported symptom.
The refutation was already written up and delivered before a human caught it.
A closely related failure: a regression test that has never been run against the broken code proves nothing. In #196 the first draft of the e2e test passed on main — it would have shipped as green certification of a live bug. Only running it against the base commit caught that. This mirrors what the issue itself faulted the old unit test for: certifying the broken path as working.
Goal
Add a Reproducing Reported Bugs subsection to standards/CLAUDE.md, adjacent to the existing Troubleshooting Failures section.
Proposed text
## Reproducing Reported Bugs
A bug report came from the real system. Your repro is a *model* of it. When the two
disagree, the model is wrong until proven otherwise.
- **Suspect the harness before the report.** Before concluding "not reproducible,"
diff the fixture against reality on every detail the alleged mechanism touches:
`curl -sI` the real endpoint and compare status *and* headers, check dtypes against
the deployed artifact, check what a stub omits versus the real API. State which
details you verified.
- **One detail decides the outcome.** A missing response header, a different dtype, or
a stubbed-away method routinely inverts the result while looking irrelevant. If you
cannot explain *why* your repro disagrees with the report, you have not finished.
- **Never report a refutation without listing what the harness reproduces**, so a
reviewer can spot the divergence you missed.
- **Lock fidelity in as an assertion, not a comment.** If the fixture must send
`cache-control: no-cache` for the suite to discriminate, assert that. A repro that
can be "simplified" into a false green eventually will be.
### Regression tests must be proven to fail
A regression test that has never run against the broken code proves nothing.
- **Run it against the base commit and watch it fail** — for the *reported* reason,
not an incidental error (a typo'd selector also "fails"). Then run it against the fix
and watch it pass. Red-then-green, both observed.
- Do this **without mutating an uncommitted working tree** — use `git show base:path`
into a temp dir, or a throwaway worktree. Never `git checkout base -- <file>` over
unbacked-up work.
- A test that passes on the base commit is worse than no test: it certifies the defect
as fixed.
Acceptance criteria
-
standards/CLAUDE.mdcontains a Reproducing Reported Bugs section with both rules (harness fidelity, prove-it-fails). - The rule is stated generically — no
AudioClassifier/PWA specifics; the 308 case may appear only as a brief illustration. - Placed near Troubleshooting Failures, whose "search for known issues first" guidance it complements rather than duplicates.
- Reachable by developers on their next
setup-envsync.
Notes
Scope is deliberately generic — nothing here is web-, JS-, or PWA-specific; the same failure mode applies to any harness modelling any production system. Repo-specific instances (e.g. the Cloudflare Pages redirect contract) are documented separately in that repo's own CLAUDE.md.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read standards/CLAUDE.md, starting with the existing Troubleshooting Failures section to choose the adjacent placement. Add the proposed generic Reproducing Reported Bugs guidance, including harness fidelity and proving regression tests fail on base. Done means all acceptance criteria are met, including setup-env reachability and no AudioClassifier or PWA-specific scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- markdown
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 90/100