anthropics / anthropics/claude-code
[BUG] Embedded CLAUDE.md intermittently reaches the API with one multibyte character replaced by three U+FFFD, changing the prompt prefix mid-session and defeating prompt caching (2.1.258, Windows)
- Vorherrschende Sprache
- Python
- Sterne
- 145k
- Forks
- 23.1k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
## Summary
In headless sessions (`claude -p`) on Windows, the text of the user's global `CLAUDE.md` that Claude Code embeds in the first user message (`messages[0].content[0].text`, inside the `` "Codebase and user instructions are shown below") intermittently arrives at the API with **one multibyte character replaced by three U+FFFD replacement characters**. The file on disk is valid UTF-8 and never changes. In the affected requests the prompt prefix therefore differs from the other requests of the same session, so every such request is a full prompt-cache miss for the whole conversation.
## Environment
- Claude Code 2.1.258 (native Windows build), Windows 11 22H2 (10.0.22621), PowerShell 7 host
- Global instructions file: `%USERPROFILE%\.claude\CLAUDE.md`, 19,597 bytes, UTF-8 without BOM, Korean prose; `bytes.decode('utf-8')` in strict mode succeeds
- Session form: `claude -p` child sessions with `--allowedTools` (used as workers), API traffic observed through a local proxy set with `ANTHROPIC_BASE_URL` that records `system`, `messages[0]` and the tool names of every outgoing request
## What I observed
Over one 20-turn session, the outgoing `messages[0].content[0].text` came in two variants. They differ in exactly one place:
- clean variant (14 of 17 requests): `'바람이 불어 창문이 흔들리는 탓에 …'`
- corrupted variant (3 of 17 requests): `'바람이 ���어 창문이 흔들리는 탓에 …'`
`difflib` opcodes between the two texts: `[('replace', 3615, 3616, 3615, 3618)]`, i.e. the single character `불` (UTF-8 `EB B6 88`) became three replacement characters. Nothing else in the block, the `system` array, or the tool list differs.
Facts about the position:
- character index 3615 of the 19,136-character block; UTF-8 byte offset 7,982 inside the block; byte offset 7,665 inside the file
- neither offset is a 4 KiB / 8 KiB / 16 KiB boundary
- the position is identical in every corrupted request across four separate sessions (two probe runs plus two long worker runs); only *whether* a given request is corrupted varies
- the corrupted variant appeared both as the very first request of a session and in later requests (for example requests 2, 11 and 32 of one session; 15, 24 and 26 of another), with the clean variant in between, and the file was not modified during any session
Per-request view from the proxy log of one session (`sid` = first 8 hex chars of `sha256(system + messages[0])`, `in` = total input tokens, `cached` = cached input tokens reported by the API):
```
msgs sid in cached
5 600ce64a 58662 0 <- first big request
7 600ce64a 63738 58659
9 600ce64a 63807 38874
11 600ce64a 68896 63735
13 600ce64a 68964 68893
15 8fdae59a 74059 0 <- corrupted variant: different prefix, full miss
17 600ce64a 74127 63804
19 600ce64a 79170 68961
21 600ce64a 79239 74124
24 8fdae59a 84380 74056 <- corrupted variant again
26 8fdae59a 84448 0 <- full miss
28 600ce64a 89476 79167
```
## Impact
Each corrupted request changes the prefix of the whole conversation, so the request misses the prompt cache for the entire context (60-100k tokens here). With the corruption on about a fifth of the requests, the sessions read 75-76% of their input from cache; with the proxy substituting the clean text for the corrupted one (a workaround, see below), the same task reads 91.6% from cache and the uncached token volume drops from about 300k to 93k. Besides cost, a request that carries `�` in the instructions is also a small fidelity loss.
## Steps to reproduce
1. Put a UTF-8 `CLAUDE.md` with a few KB of Korean (or other multibyte) prose in `%USERPROFILE%\.claude\` on Windows.
2. Point `ANTHROPIC_BASE_URL` at a local proxy that dumps `messages[0]` of every request (any pass-through proxy works).
3. Run `claude -p "" --allowedTools Read,Grep,Bash`.
4. Diff `messages[0].content[0].text` across the requests: in my runs 3 of 17 requests carried the corrupted character at the same position; the rest were byte-identical to the file.
I could not find the pattern that decides which request is corrupted; the file, its mtime and the session were unchanged between a clean and a corrupted request seconds apart.
## Related
- #40396 (closed) reported U+FFFD for Korean characters in responses on macOS; this report is about the *request* side and the embedded instructions file, on Windows.
- #88836 reports U+FFFD in AskUserQuestion since 2.1.235; possibly the same decoding path.
## Workaround
A proxy that remembers each clean first message it has seen and, when a request differs from a remembered text only by runs of U+FFFD, sends the remembered text instead. That keeps the prefix stable but should not be needed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Rechercherichtung
Reproduce with a UTF-8 Korean %USERPROFILE%\.claude\CLAUDE.md, claude -p, --allowedTools, and a proxy recording messages[0].content[0].text. Trace how the file becomes the embedded system-reminder text and compare clean and corrupted requests, checking the related U+FFFD reports in #40396 and #88836. Done means repeated requests preserve the original character and prompt-cache prefix.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Bereich
- backend, cli
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Aktiv
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 45/100