microsoft / microsoft/terminal
UTF-8 decoding problem when a codepoint straddles an i/o boundary
- Dominant language
- C++
- Stars
- 105k
- Forks
- 9.6k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 29
Description
### Windows Terminal version
1.20.11781.0
### Windows build number
10.0.19045.4780
### Other Software
_No response_
### Steps to reproduce
1. Create a text file with 180 instances of the Unicode character U+20B0 (German Penny Sign) on one line and save it as UTF-8 (with or without BOM). Call it foo.txt. (I've attached a sample.)
2. Open a Command Prompt profile in Terminal.
3. `chcp 65001`
4. `type foo.txt`
Note that, near the end of the output there are a couple Unicode replacement characters.
What's happening is that `type` sends the text to the terminal in 512-byte blocks. The UTF-8 encoding of U+20B0 takes 3 bytes. Since 512 isn't a multiple of 3, the 171st German Penny Sign is split across the boundary of the first and second write operations issued by `type`. The UTF-8 decoding is resetting state state with each write.
But it's not just UTF-8 decoding. If one write ends with a complete character, and the next write begins with a combining character, they either (1) won't be composed or (2) they will be composed but there will be an empty cell immediately after it.
These problems occur less frequently with applications that issue larger writes, but they do still happen. They can even happen with applications that normally flush the output on line boundaries if a single line grows so long that an intermediate flush occurs.
[foo.txt](https://github.com/user-attachments/files/16882048/foo.txt)
### Expected Behavior
I expected UTF-8 decoding and composition of combining characters to resync if a sequence of bytes that represents a single codepoint or grapheme cluster happens to fall on the boundary between two consecutive writes.
### Actual Behavior
Note the replacement characters in the output.

Contributor guide
Assessment
This issue has not been assessed yet.