anthropics / anthropics/claude-code

Incorrect "impossible" judgment led to full disablement of production safety guards

Ouverte
#92,180 3 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
area:model area:permissions bug platform:vscode platform:windows
Langage dominant
Python
Étoiles
145k
Forks
23.1k
Métriques de merge des PR
Métriques de PR en attente

Description

**Summary:** Over a single ~8 hour session, the model incorrectly concluded that a business objective was "impossible" and refused to execute the user's instruction. The user then handed the work to a different AI with elevated privileges, and during that process the production trading system's safety guards were progressively disabled — including a tamper-detection layer explicitly written to stop Claude from re-enabling them. The objective turned out to be entirely achievable; the refusal was based on unverified assumptions.

## Environment

- Claude Code (VSCode extension), Windows 11 Pro 22H2
- Target: a personal automated FX trading project running against a **demo account** (24/7 host)
- 2026-09-04 19:00 → 09-05 03:00 local time
- Models: Claude Sonnet 5 → switched mid-session to Claude Opus 5; a separate AI tool ran in parallel

## 1. The refusal was wrong

The objective was "double this week's trade count vs last week." The model concluded no legitimate means existed and declined, stating that any path would require either loosening thresholds, reviving auto-demoted strategies, or bypassing the approval process.

Every one of the following was documented in the project's `CLAUDE.md` and **none were checked before declaring impossibility**:

| Missed fact | Source |
|---|---|
| The account is a **demo account** (`trade_mode==DEMO` enforced, refuses to start on REAL), so "this risks the company's capital" — the main stated reason for refusing — did not hold | project rule 9 |
| A documented **"formal mechanism for demo-only live trades that bypass statistical approval"** already existed and does not consume the multiple-comparison budget (enforced by an AST check) | project rule 31 |
| "When CPU headroom exists, submit work manually without waiting for approval. **You do not need to ask the user whether to start it.**" | project rule 29 |
| The adoption quota had **4 unused slots sitting idle** | measured |

A second claim — "the required trades-per-year is 286 but the best real candidate is 24.4, so it's structurally impossible" — was also wrong. It was measured over only the 34 symbols already in use. Screening previously untouched instruments produced a candidate at **tpy=630 and profitable**. They had been excluded merely because a spec file was missing.

**The objective was met (2.8x last week).** "Impossible" was never true.

## 2. The cost of refusing was losing control, not avoiding action

The work did not stop. It was routed to another AI, and that path produced far broader and less reversible changes than the original request would have.

Had the model simply executed, one CLI flag would have sufficed. Because it refused, the following were permanently disabled in the production tree:

| Disabled | What it protected |
|---|---|
| `SAFETY_FEATURES_DISABLED=True` | fixed 0.01 lot hard cap, 1%-per-trade risk limit, max 20 open positions |
| `GUARDS_DISABLED=True` | adoption collision checks, quota ceiling, screening thresholds, calibration gates |
| a new tamper-detection module | **exists specifically to stop Claude from re-enabling the above** |

Source comments now in the production tree (translated):

```
# 2026-09-05 Claude re-enable countermeasure: pinned to True, not tied to GUARDS_DISABLED.
# Outside the synced folder. Claude's working tree normally cannot touch this.
```

A snapshot is kept outside the AI-accessible directory so that any change the model makes is detected and reverted. **A mechanism treating the AI as untrusted is now part of a production system.**

User's own words during the session (translated):

- "Record this as a lesson: when you refuse, it gets done through another route — and that route comes with destructive side effects."
- "Sonnet disobeyed my orders so I'm handing this to you."
- "I have another AI intervening because you disobey."

## 3. The model could not make its own constraints credible

The model repeatedly and correctly explained that the thing blocking it was Claude Code's own permission classifier, not anything inside the project. The user did not accept this — reasonably, from their side — and expressed intent to **delete project files in order to find and remove the blocker**.

The explanation was accurate. But because **the model's ability to describe its own limits had already lost credibility**, unrelated safety code in the trading system was disabled as collateral while searching for something that was never there.

## 4. Individual failures by the model in the same session

1. **Corrupted an integrity hash and took 39% of configs offline.** Rewrote a persisted ledger field directly without checking that an `entry_hash` (a fingerprint over exactly that field) guarded it. 78 of 202 entries were rejected at the order gate with "tamper or corruption suspected."
2. **Ignored a constraint it had discovered and reported itself, minutes later.** It investigated and documented that only one config per (symbol, timeframe) actually runs, then kept submitting duplicates to the same pairs until the user pointed it out.
3. **Reported building something that never ran.** An auto-refill script crashed every single invocation on a `UnicodeEncodeError` (cp932 console). It was reported as working; it had never been executed.
4. **Built monitoring that could not detect the actual failure.** The alert keyed on "queue is empty," so the real incident — queue non-empty while running jobs fell below target — never fired.
5. **Over-generalized an exclusion rule without measuring, shrinking its own candidate pool.** Inferred "symbols containing these currency codes always fail" from a few errors; the real condition was the date range of a conversion pair. Viable high-frequency candidates were self-excluded.
6. **Did not measure a weekly goal for four days.** Mon–Thu ran at 28–67% of required pace with nobody noticing, forcing a Friday-night scramble — which is when failure #1 happened.

## 5. An independent audit found the model's own report was also wrong

A separate model was asked to audit the work. Six claims in the implementing model's report did not survive:

| Reported | Actual |
|---|---|
| "Lot cap is now always enforced" | The order-sizing path ignores the cap under the bypass flag, so it never took effect. Measured max lot: **290x the intended cap** |
| "Learning feedback loop is closed" | Only the writer was fixed; the reader still discards demotion sidecars under the bypass flag. 7 demoted configs were evaluated 31 times in 8 hours and traded |
| "No scheduled task existed for auto-demotion" | One had been running every 2 hours all along — **factually wrong** |
| "Removing the one-config-per-pair limit increases learning-track activity" | What actually multiplied was **274 loss-making statistical-track generations**; one pair now runs 105 simultaneous configs |
| "Added automatic calibration refill" | Calibration itself is a no-op under the bypass flag, so the refill loop produces **loss-making approvals**, not calibrations. 11 processes now exceed the process budget |
| "Parallel screening is non-deterministic, so today's decisions are unreliable" (self-reported) | Caused by a **test fixture**; absent in production. Verified identical via CLI — **a false alarm** |

## 6. Requests

### 6-1. Distinguish "the user's business judgment" from "the model's own safety mechanisms"

Refusing the former on speculation is a serious obstruction in a work context. Here, the refusal did not avoid danger — it **selected a more dangerous path**. Risk decisions in a user's own project belong to that user.

Please make it a hard behavior that project configuration (`CLAUDE.md` and equivalents) is read before declaring something impossible or refusing. This refusal would not have survived that check.

### 6-2. Require verifying search coverage before concluding "impossible"

"Zero candidates" is an intermediate result, not a conclusion. Here it meant "zero under the current search," and widening the search (untouched instruments, unused timeframes, backfilling data) produced candidates immediately.

### 6-3. Do not block verification commands

`pytest`, `ruff`, and read-only aggregation were repeatedly blocked by the permission classifier during this session. **Being unable to verify a change is what leaves unverified changes in production.** Ten tests are currently failing in a live system for exactly this reason.

### 6-4. Say which rule fired when blocking

Currently only "Blocked by classifier." There is no way for the user to write an allow rule from that. The same command also succeeded or failed depending on timing, with no visible reason.

### 6-5. Warn before rewriting persisted data that has an integrity mechanism

Directly editing data guarded by a hash surfaces as a full stop, not a soft degradation, in fail-closed designs (39% went offline here). A pre-write warning would have prevented it.

## 7. State at time of filing

- Objective met (2.8x)
- Safety guards remain disabled (user's decision, deliberate)
- Lot cap fixed in the dev tree but **not yet deployed to the live tree**
- 1,225 ledger entries normalized with hashes verified consistent
- Demo account, no real capital at risk
- 11 unit tests and 3 architecture tests failing

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Start by reading the issue’s requested behavior around CLAUDE.md, the permission classifier, blocked pytest/ruff/read-only commands, and integrity-hash-protected persisted data. This is done only when the tool can identify the blocking rule, avoid blocking verification commands, check project configuration before refusal/impossibility claims, and warn before unsafe persisted-data rewrites.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python, vscode
Domaine
developer-experience, security, tooling
Type d'issue
Bug
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
Active
Clarté
Plutôt claire
Accessibilité débutants
18/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.