anthropics / anthropics/claude-code

Attached files are decoded as ISO-8859-1 and C1 control bytes are stripped, silently corrupting UTF-8 content

Open
#81,197 0 comments 0 reactions 0 assignees View on GitHub
area:ide
Dominant language
Python
Stars
145k
Forks
23.1k
PR merge metrics
PR metrics pending

Description

## Summary

When a UTF-8 file is **attached** to a conversation, its content reaches the model
mojibake-encoded. The file on disk is unaffected — only the ingested copy is corrupted.

The corruption is **lossy**, not merely a misread: 3-byte UTF-8 sequences lose their trailing
bytes and collapse to a single character. `—` (U+2014), `–` (U+2013) and `→` (U+2192) all arrive
as the same `â`, so the original character cannot be recovered — only guessed from context.

Reproduced across two separate sessions.

## Environment

- Claude Code, VS Code extension
- macOS (Darwin 25.5.0), Apple Silicon
- Shell locale: `LANG=C.UTF-8`, `LC_CTYPE=C.UTF-8`, `LC_ALL` unset
- Python in the same shell: `stdout=utf-8`, `filesystem=utf-8`, `preferred=UTF-8`

## Steps to reproduce

1. Create a UTF-8 file containing both 2-byte and 3-byte characters (see repro file below).
2. Verify on disk that it is clean UTF-8 (`·` = `c2 b7`, `→` = `e2 86 92`).
3. **Attach** that file to a message.
4. Observe the content as received.

## Expected

The model receives the file content as written: `·`, `§`, `é`, `—`, `–`, `→`.

## Actual

| char | UTF-8 bytes | received as | recoverable? |
|---|---|---|---|
| `·` U+00B7 | `c2 b7` | `·` | yes — both bytes present |
| `§` U+00A7 | `c2 a7` | `§` | yes |
| `é` U+00E9 | `c3 a9` | `é` | yes |
| `—` U+2014 | `e2 80 94` | `â` | **no** — trailing bytes gone |
| `–` U+2013 | `e2 80 93` | `â` | **no** |
| `→` U+2192 | `e2 86 92` | `â` | **no** |

## Mechanism

The observed output matches **ISO-8859-1 decoding followed by stripping C1 control characters
(U+0080–U+009F)** on all six characters tested:

```python
def simulate(ch):
latin = ch.encode('utf-8').decode('latin-1') # step 1: wrong decode
return ''.join(c for c in latin # step 2: strip C1 controls
if not (0x80 <= ord(c) <= 0x9F))

simulate('·') # '·' — matches observed
simulate('é') # 'é' — matches observed
simulate('—') # 'â' — matches observed
simulate('→') # 'â' — matches observed
```

For 2-byte sequences both bytes are printable in Latin-1, so they survive as a recoverable pair.
For 3-byte sequences the second and third bytes fall in the C1 range and are discarded, leaving
only the leading `0xE2` rendered as `â`. Since `—`, `–` and `→` share that leading byte, they
become indistinguishable.

Note that a plain Latin-1 misread alone would yield three characters (`â` + two controls), not
one. The byte loss points to a sanitising step after the decode.

## Impact

- **Silent and lossy.** Nothing warns that content was altered, and for 3-byte characters the
original cannot be reconstructed programmatically — only inferred from surrounding text. In one
document set this affected 82 characters.
- **A wrong inference reads as correct prose.** Substituting `—` where `→` belonged produces a
sentence that still parses and still reads naturally, so the error is unlikely to be noticed.
- **Non-English content is disproportionately affected.** In a Dutch-language project, ordinary
spelling (`één`, `naïeve`, `transiënt`) is corrupted in nearly every document.
- **Machine-readable conventions break.** Where a character is load-bearing — e.g. `·` as a field
separator in structured headings — the corrupted form breaks parsing rather than merely looking
wrong.

## Not the cause

- Not the source file: verified clean UTF-8 on disk before and after (`c2 b7`, `e2 86 92`).
- Not the shell environment: `C.UTF-8` throughout; files written by the session are clean UTF-8.
- Not the authoring tool: the same file, opened from disk, is byte-correct.

## Unaffected paths (workarounds)

Two other ingestion paths deliver the same file intact:

- **The Read tool** — reading the file from disk returns correct UTF-8, including 3-byte characters.
- **Pasting text directly into the message** — verified byte-identical, including `→` (`e2 86 92`).

So the practical workaround is to reference the file by path instead of attaching it.

## Repro file

Attach the file below (its contents are also shown, so the two can be compared):

```
# Encoding repro

2-byte characters (expected to survive as mojibake pairs):
- MIDDLE DOT U+00B7 = c2 b7 : ·
- SECTION SIGN U+00A7 = c2 a7 : §
- E ACUTE U+00E9 = c3 a9 : é

3-byte characters (expected to lose their trailing bytes):
- EM DASH U+2014 = e2 80 94 : —
- EN DASH U+2013 = e2 80 93 : –
- RIGHTWARDS ARROW U+2192 = e2 86 92 : →

Heading with a normative separator:
### ID · label · subject
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the attachment ingestion path with the supplied UTF-8 characters, then trace that path while comparing it with the unaffected Read tool and paste paths. Done means attached content preserves the original characters, including the three-byte examples, without silent loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.