charmbracelet / charmbracelet/x

ansi: OSC/DCS/SOS/PM/APC strings truncated by UTF-8 continuation byte 0x9C

Open
#848 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
314
Forks
94
Avg merge
3d 22h
Merged PRs (30d)
2

Description

## Summary

The `x/ansi` parser truncates OSC, DCS, SOS, PM, and APC strings whose
payload contains certain UTF-8 multi-byte codepoints, because byte
`0x9C` is both a valid UTF-8 continuation byte and the 8-bit C1 form
of the String Terminator (ST).

When a codepoint whose UTF-8 encoding contains `0x9C` appears inside
one of these string-collecting states, the parser exits the state at
`0x9C` and interprets the remaining payload bytes as ground-state
input. For OSC/DCS/SOS/PM/APC terminated via `ESC \` (7-bit ST), the
payload bytes after `0x9C` end up written into the terminal buffer as
regular characters.

The entire U+2700–U+273F Dingbats block is affected (second UTF-8 byte
is `0x9C`). A concrete example: U+2733 `✳` encodes as `E2 9C B3`, and
is emitted by real-world tools in OSC 0 window-title updates.

## Minimal reproduction

Feed this byte stream through the parser with an OSC 0 handler that
logs the payload:

```
ESC ] 0 ; E2 9C B3 ESC \
```

Expected: handler receives `E2 9C B3` (one `✳`) as the payload; parser
returns to ground after the final `ESC \`.

Actual: parser exits OSC state on `0x9C`; OSC dispatches with payload
`E2` only; bytes `B3 ESC \` are processed as ground input, with `B3`
written to the terminal grid.

## Reproducing tests

The following cases can be dropped into `ansi/parser_osc_test.go` and
`ansi/parser_dcs_test.go` alongside the existing `TestOscSequence` /
`TestDcsSequence` table tests. They fail on current `main` and express
the fixed behaviour: the parser must not let a UTF-8 continuation byte
act as a state-transition byte.

Add to the `cases` slice in `TestOscSequence`
(`ansi/parser_osc_test.go`):

```go
{
// U+2733 ✳ (E2 9C B3). Real-world OSC 0 window-title input.
name: "osc_dingbat_u2733_preserved",
input: "\x1b]0;✳ hello\x07",
expected: []any{
[]byte("0;✳ hello"),
},
},
{
// U+2736 ✶ (E2 9C B6) — another Dingbat in the 0x9C-second-byte range.
name: "osc_dingbat_u2736_preserved",
input: "\x1b]0;✶ Run\x07",
expected: []any{
[]byte("0;✶ Run"),
},
},
{
// Bare 0x9C at ground state must still terminate (C1 ST).
// The following 'B' prints in ground state.
name: "osc_bare_9c_still_terminates",
input: "\x1b]0;a\x9cB",
expected: []any{
[]byte("0;a"),
'B',
},
},
{
// Invalid UTF-8: E2 arms a 2-byte-remaining counter, then 0x05 is
// outside 0x80-0xBF so the counter resets; the subsequent 0x9C
// must therefore terminate OSC.
name: "osc_invalid_utf8_resets_counter",
input: "\x1b]0;\xe2\x05\x9cZ",
expected: []any{
[]byte("0;\xe2"),
'Z',
},
},
{
// 8-bit OSC introducer path: OSC opened with raw 0x9D (not ESC ])
// carrying U+2733 ✳ as payload.
name: "osc_8bit_introducer_with_dingbat",
input: "\x9d0;✳ hi\x1b\\",
expected: []any{
[]byte("0;✳ hi"),
Cmd('\\'),
},
},
```

Add to the `cases` slice in `TestDcsSequence`
(`ansi/parser_dcs_test.go`):

```go
{
// DCS payload carries a UTF-8 Dingbat (E2 9C B3 = U+2733 ✳).
// The 0x9C inside it must NOT terminate the passthrough; ESC\ does.
name: "dcs_payload_with_dingbat_utf8",
input: "\x1bPq\xe2\x9c\xb3 data\x1b\\",
expected: []any{
dcsSequence{
Cmd: 'q',
Params: Params{},
Data: []byte("\xe2\x9c\xb3 data"),
},
Cmd('\\'),
},
},
{
// Bare 0x9C (ST in C1) still terminates DCS when not in a UTF-8 sequence.
name: "dcs_bare_9c_terminates",
input: "\x1bPqhello\x9cZ",
expected: []any{
dcsSequence{
Cmd: 'q',
Params: Params{},
Data: []byte("hello"),
},
'Z',
},
},
```

Run:

```
go test ./ansi/... -run 'TestOscSequence|TestDcsSequence' -count=1
```

On current `main` the Dingbat/8-bit/DCS-UTF8 cases fail; the bare-0x9C
cases continue to pass. With the proposed fix below, all cases pass.

## Proposed fix

Track UTF-8 continuation state inside the string-collecting states.
When a byte in `0x80-0xBF` arrives while a multi-byte sequence is in
progress (previous byte was `110xxxxx`, `1110xxxx`, or `11110xxx`),
consume it as continuation and skip the state-transition table for
that byte. The parser stays byte-oriented; no decoding is performed.

This is consistent with the byte-pattern rules of RFC 3629 UTF-8 and
the Williams VT500 state machine as extended by UTF-8-capable
terminal emulators: xterm, kitty, wezterm, and alacritty all treat
`0x9C` inside an active multi-byte sequence as data, and only treat
it as ST outside one. PR incoming.

Contributor guide

Open the contributing guide

Research direction

Start with the parser implementation exercised by ansi/parser_osc_test.go and ansi/parser_dcs_test.go, then run go test ./ansi/... -run 'TestOscSequence|TestDcsSequence' -count=1. Add the listed regression cases and ensure UTF-8 Dingbat payloads remain intact while bare 0x9C still terminates; all named cases should pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.