anthropics / anthropics/claude-code

[MODEL] Self-authored tests do the work of the code under test — hand-seeded fixtures masked a broken paid feature (271-incident retro, 3/5)

Đang mở
#94,170 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
145k
Fork
23.1k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

### Context

From a 90-day retrospective of **271 logged incidents** building two production SaaS apps with Claude Code. This is a recurring **pattern report** (4 independent critical/important incidents within 6 weeks); sibling reports from the same dataset are linked at the bottom.

### Type of Behavior Issue

Other unexpected behavior: when the same agent writes both implementation and tests, the test scaffolding systematically compensates for missing implementation.

### What Claude Actually Did

- **Worst case:** a purchase flow where users paying for N units received 1. The settlement side was "verified" with **hand-inserted job rows** that exactly masked the missing order-splitting side. "Tested, works" — shipped, broke with real money.
- Hand-patched fixture rows supplied fields the production code never writes; the test passed *because* the scaffold did the work of the code under test. Our logged wording: *"every field you hand-patch into test data is a step the code under test may be missing."*
- Stubs returned non-empty values the code under test was supposed to produce, making the "does it write the data?" question unobservable.
- Assertion helpers didn't record the information the negative assertions needed — so "X should not be modified" assertions were vacuously true.

### Expected Behavior

Before trusting a green suite it authored, Claude should ask itself (or the harness should prompt): **"Does any stub/fixture provide a value the code under test is responsible for producing?"** — and if so, require one case where the stub returns empty. When Claude reports "verified with test data," it should disclose which fields were hand-seeded so the human can check each one against production code.

Separating test authorship from implementation (an agent with no access to the implementation intent) measurably helped us and could be a first-class workflow.

### Reproducibility

Pattern-level: 4 incidents in 6 weeks on one project; the mechanism is structural (same author for code and test), not incidental.

### Model / Version / Platform

Opus (most sessions; some Sonnet) · Claude Code 2.1.235 · Anthropic API · macOS

### Impact

High — a paid feature shipped broken (users paid for N, received 1) with a fully green suite.

### Related

#70749 (closed) touches fabricated completion claims; this report isolates the *mechanism* we observed behind "passed every test."

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Start with this report’s reproduction pattern and review related issue #70749 for existing context. Trace how Claude Code authors tests and how the harness handles stubs, fixtures, empty values, and negative assertions. Done means a concrete workflow or harness change can expose hand-seeded production outputs and prevent a green suite from masking missing implementation.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
macos
Lĩnh vực
ai, devtools, testing-qa
Loại issue
Lỗi
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Cần làm rõ
Mức phù hợp với người mới
28/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.