TimZander / TimZander/claude

standards/CLAUDE.md: reproducing reported bugs — suspect the harness, and prove the test fails on base

Open Beginner friendly
#184 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
1
Avg merge
1d 3h
Merged PRs (30d)
7

Description

Problem

A repro harness that diverges from production on a single detail produces a confident false negative — the bug looks refuted, and the report gets closed or re-litigated from a wrong premise. This has now produced a wrong answer four times across recent sessions in AudioClassifier:

Case The one detail that diverged
_headers vs. the wire Local server didn't send what production sends
FP32 vs. FP16 Harness dtype differed from the deployed model
Stubbed AbortSignal Stub omitted behavior the real API has
#196 (net::ERR_FAILED offline) Test server sent a bare 308, production sends 308 + cache-control: no-cache

The #196 case is the clearest. The claim was that the installed PWA can't launch offline because cache.addAll follows a 308 and precaches a redirected: true response, which respondWith rejects for a navigation. The harness sent a bare 308 and the app launched fine — reported as not reproducible. That was wrong: a bare 308 is a permanent redirect and therefore heuristically cacheable, so Chromium cached it and replayed /index.html/ offline without ever consulting the service worker, landing on a clean cache entry. Production sends cache-control: no-cache, which defeats the replay, so the navigation reaches the SW and hits the poisoned entry. Adding that one header made the test fail on main with the exact reported symptom.

The refutation was already written up and delivered before a human caught it.

A closely related failure: a regression test that has never been run against the broken code proves nothing. In #196 the first draft of the e2e test passed on main — it would have shipped as green certification of a live bug. Only running it against the base commit caught that. This mirrors what the issue itself faulted the old unit test for: certifying the broken path as working.

Goal

Add a Reproducing Reported Bugs subsection to standards/CLAUDE.md, adjacent to the existing Troubleshooting Failures section.

Proposed text

## Reproducing Reported Bugs

A bug report came from the real system. Your repro is a *model* of it. When the two
disagree, the model is wrong until proven otherwise.

- **Suspect the harness before the report.** Before concluding "not reproducible,"
  diff the fixture against reality on every detail the alleged mechanism touches:
  `curl -sI` the real endpoint and compare status *and* headers, check dtypes against
  the deployed artifact, check what a stub omits versus the real API. State which
  details you verified.
- **One detail decides the outcome.** A missing response header, a different dtype, or
  a stubbed-away method routinely inverts the result while looking irrelevant. If you
  cannot explain *why* your repro disagrees with the report, you have not finished.
- **Never report a refutation without listing what the harness reproduces**, so a
  reviewer can spot the divergence you missed.
- **Lock fidelity in as an assertion, not a comment.** If the fixture must send
  `cache-control: no-cache` for the suite to discriminate, assert that. A repro that
  can be "simplified" into a false green eventually will be.

### Regression tests must be proven to fail

A regression test that has never run against the broken code proves nothing.

- **Run it against the base commit and watch it fail** — for the *reported* reason,
  not an incidental error (a typo'd selector also "fails"). Then run it against the fix
  and watch it pass. Red-then-green, both observed.
- Do this **without mutating an uncommitted working tree** — use `git show base:path`
  into a temp dir, or a throwaway worktree. Never `git checkout base -- <file>` over
  unbacked-up work.
- A test that passes on the base commit is worse than no test: it certifies the defect
  as fixed.

Acceptance criteria

  • standards/CLAUDE.md contains a Reproducing Reported Bugs section with both rules (harness fidelity, prove-it-fails).
  • The rule is stated generically — no AudioClassifier/PWA specifics; the 308 case may appear only as a brief illustration.
  • Placed near Troubleshooting Failures, whose "search for known issues first" guidance it complements rather than duplicates.
  • Reachable by developers on their next setup-env sync.

Notes

Scope is deliberately generic — nothing here is web-, JS-, or PWA-specific; the same failure mode applies to any harness modelling any production system. Repo-specific instances (e.g. the Cloudflare Pages redirect contract) are documented separately in that repo's own CLAUDE.md.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read standards/CLAUDE.md, starting with the existing Troubleshooting Failures section to choose the adjacent placement. Add the proposed generic Reproducing Reported Bugs guidance, including harness fidelity and proving regression tests fail on base. Done means all acceptance criteria are met, including setup-env reachability and no AudioClassifier or PWA-specific scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
markdown
Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
90/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.