cactus-compute / cactus-compute/cactus
Non-ASCII input corrupted: Python binding escapes to \uXXXX (ensure_ascii) and the engine never decodes it
- Dominant language
- C++
- Stars
- 6k
- Forks
- 503
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 4
Description
## Summary
Non-ASCII input (accented Latin, Japanese, Korean, …) is corrupted before it reaches the model when using the Python bindings / `cactus serve` (v2.1.0, macOS ARM64). Severity scales with multi-byte character density: English is unaffected, German nearly so, French partially broken, Japanese/Korean unusable.
## Root cause
`python/cactus/bindings/cactus.py::_to_json` marshals messages/tools with `json.dumps(obj)`. Python's default `ensure_ascii=True` rewrites every non-ASCII character as `\uXXXX`, and the engine's messages parser does not decode `\u` escapes: `cactus-engine/src/engine.h::extract_json_string` (line ~305) handles only `\n \t \r \" \\` and falls through to `value += json[pos]` for anything else, so `\u660e` becomes the literal text **`u660e`** — the model receives `u660eu65e5u306e…` for `明日の`. Each escaped character costs ~3 tokens and is semantic noise.
## Evidence (model-independent)
Same messages+tools through the model's HF tokenizer vs `usage.prompt_tokens` from `cactus serve` (prebuilt-graph bundle of LiquidAI/LFM2.5-230M):
| lang | HF tokens | cactus tokens | excess beyond the constant en offset (+42) |
|---|---:|---:|---:|
| en | 400 | 442 | 0 |
| fr | 409 | 467 | +16 |
| ko | 408 | 495 | +45 |
| ja | 403 | 527 | +82 |
A Python replica of `bpe.cpp` (its byte map + the bundle's vocab/merges + whole-segment BPE) matches HF exactly for en/fr, so the tokenizer algorithm itself is fine — the corruption happens upstream of `encode()`.
**With `ensure_ascii=False` the excess is a constant +42 in all four languages** (en/fr/ja/ko: +42/+42/+44/+42), and on an 808-row multilingual tool-calling eval, ja/ko false-action rates drop from 88–100% back to the bf16 reference (0–16%).
## 10-second repro
POST any Japanese sentence to `/v1/chat/completions` and compare `usage.prompt_tokens` with `len(AutoTokenizer.from_pretrained()(text).input_ids)`.
## Fix
One line in `_to_json`: `json.dumps(obj, ensure_ascii=False).encode()` — **PR: #804** (https://github.com/cactus-compute/cactus/pull/804). Alternative/complementary: have the C++ messages JSON reader decode `\uXXXX` — `utils.h` already contains a correct decoder used elsewhere.
## Second, minor bug found on the way (C++, separate) — now tracked in #806, fixed by #805
`bpe.cpp::init_byte_mappings` maps bytes 161..255 to identity code points; GPT-2 byte-level BPE excludes 173 (U+00AD), which must map to U+0143 like the other non-printables. UTF-8 sequences containing byte 0xAD tokenize differently from the reference (a Japanese test sentence: 17 tokens with 1 unknown vs 15 in HF). Concrete symptom: `歯` is `e6 ad af`; `cactus run` on the stock `lfm2.5-230m-cq4` bundle echoes it as `�丈` (U+FFFD + wrong kanji) — this mapping bug on the decode side, independent of the escaping bug above. Fix: exclude 173 from the identity byte list AND from the identity branch of the assignment loop (`else if (byte >= 161 && byte <= 255 && byte != 173)`) — the branch is chosen by byte value, so moving the byte between lists alone is a no-op — and append 173 to `remaining_bytes` after 160 (→ U+0143). Verified: `cactus_tokenize("歯")` → `[33687]` (HF reference) after the change; before: `[672, 0, 617]`. **PR: https://github.com/cactus-compute/cactus/pull/805**
Related (decode side, Gemma): #684.
Environment: cactus 2.1.0 (brew + pip `cactus-compute[convert]`), macOS ARM64 (Darwin 25.4.0).
Contributor guide
Research direction
Start in python/cactus/bindings/cactus.py at _to_json and inspect cactus-engine/src/engine.h::extract_json_string; keep the separate bpe.cpp issue out of scope. Reproduce the Japanese or Korean request through cactus serve, then verify prompt-token counts and multilingual tool-calling behavior against the stated references once the input reaches the model intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100