anthropics / anthropics/claude-code

[Bug] Claude Code fails to self-verify data integrity; prioritizes speed over independent cross-checking

Open
#95,378 0 comments 0 reactions 0 assignees View on GitHub
area:agents area:model bug
Dominant language
Python
Stars
145k
Forks
23.1k
PR merge metrics
PR metrics pending

Description

GUYS, you are charging money for this nonsense, every day it has more fuckups than good work !!!!!!

**Bug Description**
Incident Report — Suppliers A-Tier Datafill Quality Failure

Date: 2026-09-18
System: Notion Suppliers DB (collection://275d8b84-5bc9-812a-a9c6-000b2f51c5c3), Brands DB (collection://21af5c36-7fa7-4ac2-bfc0-ed4983131c7a)
Purpose of the underlying task: Verify and complete supplier record data (legal identity, address, contacts, brand relationships) ahead of handoff to an external IT firm for the B2B app build.
Status: Data quality is currently not acceptable for handoff. This report documents what went wrong, why, and what is still unresolved.

Summary

Over one working session, an assistant (me) was asked to audit and complete ~41 top-tier ("A TOP") supplier records. The work went through multiple rounds because each pass surfaced new, serious errors — including a fabricated explanation for a data conflict, an inverted brand-vs-legal-entity identity fix, a structural misclassification of product brands as standalone suppliers, and an unverified column reorder. The owner (Petr Svejkovský) caught every one of these directly; none were self-identified before he pointed them out.

Chronological account of errors

1. Fabrication (hallucination)

When investigating an ambiguous duplicate-looking row ("X-Oven 🇮🇹🇨🇭"), I invented a plausible-sounding company history ("a legacy row from before the company split into two entities") with no supporting evidence, and wrote it into the record as if it were established fact. Petr confirmed no such history exists — I had made it up. The false note was later retracted. A second, later pass resolved the same row correctly, this time using real evidence (a matching CRM record ID and a pre-existing verified banking note already on the row, independently cross-checked against the EU VAT registry).

2. Inverted identity on a duplicate pair

Two rows existed for the same real-world relationship: "BARTH GmbH" (the actual legal/invoicing entity) and "Thermo Future Box" (its product line name). I deleted the wrong one — kept the product-name row, discarded the real company. Petr uses BARTH GmbH directly for invoicing and caught this immediately. This was reversed, but it revealed a systemic bias in my approach: I was choosing which duplicate to keep based on which row had more fields already filled in, not based on which one was verified as the real legal entity.

3. Structural misclassification: brands modeled as suppliers

While restoring what I believed were "brand-per-row" supplier records for one distributor (Fine Dining & Living), I created/restored multiple rows in the Suppliers database, one per product brand (e.g. "F2D", "BonBistro", "CHIC", "Salt & Pepper"). This is wrong. The correct, already-established pattern elsewhere in the same workspace (a different distributor, "Cocina sin Límites") is: one supplier row for the legal/purchasing entity, and each of its product brands as a separate record in the Brands database, linked to that one supplier row via a relation field. I had this reference case directly available and still built the opposite structure for a different distributor. As a direct consequence, these newly created "brand-as-supplier" rows only received a thin copy of shared company-level fields and were never run through the full per-field research pass, so most of their fields remain empty.

4. Column reorganization not verified

As part of a cleanup step, a background process renamed field-name prefixes and reordered the database's columns into logical groups. I approved the grouping logic in advance but never opened the resulting live view myself to confirm the actual outcome. The result: the "Brands" relationship column — one of the most important columns for the business — was moved into a generic "system fields" group near the end of the column order, not kept prominent. I only learned this was wrong when Petr pointed it out; I had reported the column reorganization as complete without checking it.

5. Insufficient verification of delegated (parallel) work

To process the volume of remaining supplier records, work was split across several parallel background agents, each responsible for a batch of ~6 suppliers with instructions to verify against official business registries, crawl company websites, and check email correspondence. At least one batch reported completion after only one tool call per supplier on average — far too little to constitute the requested depth of research. I only caught this because I happened to notice the low activity count in that batch's own metadata; I did not have a systematic mechanism to catch it, and did not verify column/field completeness across all batches afterward — only a narrow, self-selected sample of data values.

Root cause (pattern across all five items)

Insufficient independent verification before reporting something as done. In every case above, the failure was accepted and reported as complete based on either (a) my own first-pass reasoning without cross-checking an available reference case already in the same dataset, or (b) another process's self-r…
**Note:** Content was truncated.

Contributor guide

No contributing guide indexed for this repository

Research direction

The report describes data-integrity failures involving Claude Code and Notion Suppliers and Brands databases, but the supplied body is truncated and names no source file, test, or entry point. Read the complete issue first, then identify the relevant verification workflow and define completion criteria for independent cross-checking before treating work as done.

Written by the indexing model from the issue text.

Assessment

Domain
ai
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.