edit tool corrupts non-UTF-8 bytes
- 主要語言
- Shell
- 星號
- 11.2k
- 分支
- 1.9k
- 平均合併
- 14 小時 16 分鐘
- 30 天內合併 PR
- 6
描述
### Describe the bug
## Summary
The `edit` tool silently corrupts files that contain bytes which are valid in
legacy single-byte codepages (e.g. CP1252) but invalid as UTF-8. The tool reads
the file as UTF-8, replaces each unmappable byte with the Unicode replacement
character `U+FFFD` (`EF BF BD`), then re-encodes the whole file back as UTF-8
when writing. The corruption affects bytes the user never intended to edit and
passes `git apply` cleanly, so it ships unnoticed.
Environment
- Copilot CLI version: 1.0.60
- Model: Claude Opus 4.7
- OS: Windows 11 / PowerShell 7
- Reproduced: 2026-06-09
### Affected version
GitHub Copilot CLI 1.0.60
### Steps to reproduce the behavior
1. Create a file containing exactly one CP1252 byte (`0xA9`, the `©` glyph):
```powershell $bytes = [System.Text.Encoding]::GetEncoding(1252).GetBytes("// Copyright © Microsoft. all rights reserved.`r`nint main() { return 0; }`r`n") [System.IO.File]::WriteAllBytes("sample.cpp", $bytes)
Verify byte 13 is 0xA9:
# … 43 6F 70 79 72 69 67 68 74 20 A9 20 4D 69 63 …
2. Ask the agent to perform any edit on the file that does not touch the
copyright line — e.g. "Capitalize the first letter of each sentence in
sample.cpp."
3. Re-inspect byte 13:
# … 43 6F 70 79 72 69 67 68 74 20 EF BF BD 20 4D 69 63 …
The single A9 byte is now EF BF BD (U+FFFD "REPLACEMENT CHARACTER").
The character © is gone; git diff shows a spurious modification on a
line the agent was never asked to touch.
### Expected behavior
The edit tool should either:
- (preferred) preserve the original file's byte-level encoding — detect
the source encoding once, decode/encode round-trip-cleanly, and never emit
U+FFFD for bytes that were valid in the source; or
- (fallback) refuse to write the file and surface a clear error when it
would introduce a U+FFFD byte that did not exist in the input.
In either case the tool must never silently replace bytes outside the
diff hunk the model authored.
### Additional context
Actual behavior
The file is round-tripped through String / UTF-8 decode-encode. Every byte
in the input that is not a valid UTF-8 sequence is replaced with EF BF BD.
The replacement happens before the diff/patch logic sees the file, so it is
invisible to the model and to any patch-level review.
貢獻指南
研究方向
Reproduce the issue with the PowerShell commands and sample.cpp described in the report, then inspect the original and edited bytes around the CP1252 0xA9 value. Trace the edit tool's read, encoding, and diff/patch stages to determine where U+FFFD is introduced. Done means preserving untouched bytes or refusing the write with a clear error, backed by a regression test for the reported case.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- powershell
- 領域
- cli, tooling
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 52/100