cactus-compute / cactus-compute/cactus

Non-ASCII input corrupted: Python binding escapes to \uXXXX (ensure_ascii) and the engine never decodes it

Open
#803 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6k
Forks
503
Avg merge
1d 18h
Merged PRs (30d)
4

Description

## Summary
Non-ASCII input (accented Latin, Japanese, Korean, …) is corrupted before it reaches the model when using the Python bindings / `cactus serve` (v2.1.0, macOS ARM64). Severity scales with multi-byte character density: English is unaffected, German nearly so, French partially broken, Japanese/Korean unusable.

## Root cause
`python/cactus/bindings/cactus.py::_to_json` marshals messages/tools with `json.dumps(obj)`. Python's default `ensure_ascii=True` rewrites every non-ASCII character as `\uXXXX`, and the engine's messages parser does not decode `\u` escapes: `cactus-engine/src/engine.h::extract_json_string` (line ~305) handles only `\n \t \r \" \\` and falls through to `value += json[pos]` for anything else, so `\u660e` becomes the literal text **`u660e`** — the model receives `u660eu65e5u306e…` for `明日の`. Each escaped character costs ~3 tokens and is semantic noise.

## Evidence (model-independent)
Same messages+tools through the model's HF tokenizer vs `usage.prompt_tokens` from `cactus serve` (prebuilt-graph bundle of LiquidAI/LFM2.5-230M):

| lang | HF tokens | cactus tokens | excess beyond the constant en offset (+42) |
|---|---:|---:|---:|
| en | 400 | 442 | 0 |
| fr | 409 | 467 | +16 |
| ko | 408 | 495 | +45 |
| ja | 403 | 527 | +82 |

A Python replica of `bpe.cpp` (its byte map + the bundle's vocab/merges + whole-segment BPE) matches HF exactly for en/fr, so the tokenizer algorithm itself is fine — the corruption happens upstream of `encode()`.

**With `ensure_ascii=False` the excess is a constant +42 in all four languages** (en/fr/ja/ko: +42/+42/+44/+42), and on an 808-row multilingual tool-calling eval, ja/ko false-action rates drop from 88–100% back to the bf16 reference (0–16%).

## 10-second repro
POST any Japanese sentence to `/v1/chat/completions` and compare `usage.prompt_tokens` with `len(AutoTokenizer.from_pretrained()(text).input_ids)`.

## Fix
One line in `_to_json`: `json.dumps(obj, ensure_ascii=False).encode()` — **PR: #804** (https://github.com/cactus-compute/cactus/pull/804). Alternative/complementary: have the C++ messages JSON reader decode `\uXXXX` — `utils.h` already contains a correct decoder used elsewhere.

## Second, minor bug found on the way (C++, separate) — now tracked in #806, fixed by #805
`bpe.cpp::init_byte_mappings` maps bytes 161..255 to identity code points; GPT-2 byte-level BPE excludes 173 (U+00AD), which must map to U+0143 like the other non-printables. UTF-8 sequences containing byte 0xAD tokenize differently from the reference (a Japanese test sentence: 17 tokens with 1 unknown vs 15 in HF). Concrete symptom: `歯` is `e6 ad af`; `cactus run` on the stock `lfm2.5-230m-cq4` bundle echoes it as `�丈` (U+FFFD + wrong kanji) — this mapping bug on the decode side, independent of the escaping bug above. Fix: exclude 173 from the identity byte list AND from the identity branch of the assignment loop (`else if (byte >= 161 && byte <= 255 && byte != 173)`) — the branch is chosen by byte value, so moving the byte between lists alone is a no-op — and append 173 to `remaining_bytes` after 160 (→ U+0143). Verified: `cactus_tokenize("歯")` → `[33687]` (HF reference) after the change; before: `[672, 0, 617]`. **PR: https://github.com/cactus-compute/cactus/pull/805**

Related (decode side, Gemma): #684.

Environment: cactus 2.1.0 (brew + pip `cactus-compute[convert]`), macOS ARM64 (Darwin 25.4.0).

Contributor guide

Open the contributing guide

Research direction

Start in python/cactus/bindings/cactus.py at _to_json and inspect cactus-engine/src/engine.h::extract_json_string; keep the separate bpe.cpp issue out of scope. Reproduce the Japanese or Korean request through cactus serve, then verify prompt-token counts and multilingual tool-calling behavior against the stated references once the input reaches the model intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.