cactus-compute / cactus-compute/cactus

Gemma byte-fallback tokens are not reassembled into UTF-8

Open
#684 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6k
Forks
501
Avg merge
1d 18h
Merged PRs (30d)
4

Description

## Summary

Gemma byte-fallback tokens such as `<0xEA>` can surface as literal text instead of being reassembled into UTF-8 during detokenization.

This causes valid generated text in languages such as Korean and Polish to appear with raw byte-fallback pieces in the final output.

## Example

Observed Gemma 4 audio transcription output:

잠 <0xEA><0xB9><0xB0> 수 있는 영상 틀어 줄래?

The contiguous byte span is valid UTF-8 and should decode to `깰`, so the intended output is:

잠 깰 수 있는 영상 틀어 줄래?

## Why This Matters

Gemma-family tokenizers can emit byte-fallback pieces when a codepoint is not represented as a single vocabulary piece. If the detokenizer does not reassemble contiguous `<0xHH>` byte pieces, downstream callers see literal byte markers instead of user-facing text.

This affects Korean and other non-ASCII languages, and it can also affect mixed-language output when byte fallback is used.

## Expected Behavior

When `byte_fallback` is enabled, contiguous runs of `<0xHH>` pieces should be decoded as UTF-8 if they form a valid sequence.

Invalid or incomplete byte runs should remain visible as literal `<0xHH>` text so truncated or malformed output is not silently dropped.

## Proposed Fix

Add a shared tokenizer helper that:

- parses `<0xHH>` byte-fallback pieces,
- reassembles contiguous byte-fallback runs into UTF-8 when valid,
- preserves invalid or incomplete runs unchanged,
- applies this post-processing in both SP and BPE decode paths that can return aggregated text,
- preserves streaming behavior for single-token byte-fallback decode.

A PR implementing this is already open here:

https://github.com/cactus-compute/cactus/pull/635

## Validation From The PR

The PR adds `tests/test_byte_fallback_detokenize.cpp` with cases for:

- valid uppercase/lowercase byte fallback pieces,
- malformed pieces,
- Korean 3-byte reassembly,
- Polish 2-byte reassembly,
- ASCII byte fallback,
- invalid partial spans,
- trailing incomplete bytes,
- multiple byte-fallback runs separated by normal text.

On Android arm64-v8a / Galaxy S10e, the added byte-fallback test executable passed 10/10 cases.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing PR #635 and tests/test_byte_fallback_detokenize.cpp, then trace the SP and BPE decode paths that return aggregated text. Done means the listed valid fallback cases decode to UTF-8, malformed or incomplete runs remain literal, and single-token streaming behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.