cactus-compute / cactus-compute/cactus
Gemma byte-fallback tokens are not reassembled into UTF-8
- Dominant language
- C++
- Stars
- 6k
- Forks
- 501
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 4
Description
## Summary
Gemma byte-fallback tokens such as `<0xEA>` can surface as literal text instead of being reassembled into UTF-8 during detokenization.
This causes valid generated text in languages such as Korean and Polish to appear with raw byte-fallback pieces in the final output.
## Example
Observed Gemma 4 audio transcription output:
잠 <0xEA><0xB9><0xB0> 수 있는 영상 틀어 줄래?
The contiguous byte span is valid UTF-8 and should decode to `깰`, so the intended output is:
잠 깰 수 있는 영상 틀어 줄래?
## Why This Matters
Gemma-family tokenizers can emit byte-fallback pieces when a codepoint is not represented as a single vocabulary piece. If the detokenizer does not reassemble contiguous `<0xHH>` byte pieces, downstream callers see literal byte markers instead of user-facing text.
This affects Korean and other non-ASCII languages, and it can also affect mixed-language output when byte fallback is used.
## Expected Behavior
When `byte_fallback` is enabled, contiguous runs of `<0xHH>` pieces should be decoded as UTF-8 if they form a valid sequence.
Invalid or incomplete byte runs should remain visible as literal `<0xHH>` text so truncated or malformed output is not silently dropped.
## Proposed Fix
Add a shared tokenizer helper that:
- parses `<0xHH>` byte-fallback pieces,
- reassembles contiguous byte-fallback runs into UTF-8 when valid,
- preserves invalid or incomplete runs unchanged,
- applies this post-processing in both SP and BPE decode paths that can return aggregated text,
- preserves streaming behavior for single-token byte-fallback decode.
A PR implementing this is already open here:
https://github.com/cactus-compute/cactus/pull/635
## Validation From The PR
The PR adds `tests/test_byte_fallback_detokenize.cpp` with cases for:
- valid uppercase/lowercase byte fallback pieces,
- malformed pieces,
- Korean 3-byte reassembly,
- Polish 2-byte reassembly,
- ASCII byte fallback,
- invalid partial spans,
- trailing incomplete bytes,
- multiple byte-fallback runs separated by normal text.
On Android arm64-v8a / Galaxy S10e, the added byte-fallback test executable passed 10/10 cases.
Contributor guide
Research direction
Start by reviewing PR #635 and tests/test_byte_fallback_detokenize.cpp, then trace the SP and BPE decode paths that return aggregated text. Done means the listed valid fallback cases decode to UTF-8, malformed or incomplete runs remain literal, and single-token streaming behavior is preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100