Multilingual byte-divergence from HuggingFace Python: 4 distinct bugs across WordPiece / Unigram / BPE-byte-fallback

# Multilingual byte-divergence: 4 distinct bugs across WordPiece / Unigram / BPE-byte-fallback

## How these bugs were found

I worked with Anthropic's Claude Code (Opus 4.7 1M extra high) building an Apple-ecosystem Obj-C port of `Tokenizers` (published at (https://github.com/apocryphx/ObjCTokenizer)), originally for byte-identical correctness in embedding pipelines on Apple Silicon. As part of bringing the port up to publication readiness we added a multilingual diagnostic test corpus alongside the existing English byte-identical baseline. The new corpus is 67 lines covering CJK simplified + traditional, Japanese, Korean Hangul, Arabic, Hebrew, Devanagari, Thai, mixed-script with emoji separators, BMP + astral-plane glyphs, programming code, and IPA — chosen to stress axes the English-only corpus doesn't reach. Result: three of five tested kernels failed against HF Python reference (`transformers` 4.57.x): BGE-small WordPiece 8/67 failed, T5-small Unigram 1/67, Llama-7B BPE byte-fallback 6/67. Byte-level BPE families (GPT-2, RoBERTa-base) passed cleanly.

To determine whether the divergences were introduced by porting or inherited from upstream, we built a small Swift CLI harness loading `Tokenizers.AutoTokenizer.from(tokenizerConfig:tokenizerData:)` directly from swift-transformers `main` at commit `349a7ce` and ran the same comparison. Result: swift-transformers itself diverged on the same lines with near-identical pass/fail counts (BGE 58/67); T5 66/67 identical; Llama-7B 61/67 identical). Our conclusion: the bugs are inherited from this codebase, not introduced by porting choices.

Each failing line was reduced to a 1-line minimal repro and the affected pipelines instrumented at multiple sites. For Bug 4 specifically, comparing the post-`bpe(token:)` pieces array to what arrived at the vocab-lookup site surfaced the `componentsSeparatedByString:` grapheme-cluster issue, which wouldn't have been visible from single-end probing. The four root causes laid out in the per-bug sections fell out cleanly from those probes.

Claude Code created corpus design, the swift-transformers comparison probe, multi-site instrumentation, architectural-pattern recognition that unified the four bugs under one root cause family (Cocoa's grapheme-cluster awareness silently misbehaving at vocab-lookup boundaries), and fix implementation across both kernel-space and helper-space code.

The result is independently verifiable: the corpus is plain text, the goldens are produced by HuggingFace Python via the open-source `make golden` pipeline shipped in the port, the Swift probe is reproducible via `swift run`, and each bug has a 1-line repro evaluable by inspection. None of the verification chain depends on running an AI model — maintainers can check everything end-to-end with `transformers` and `swift build`.

I’m naming this collaboration because using AI effectively on correctness-critical tasks, like tokenizer byte-identity, is truly effective when the approach ensures verifiability throughout.

ObjCTokenizer is the only Obj-C/Swift HuggingFace tokenizer port I'm aware of that's byte-identical with HF Python on multilingual input across this kernel set. The diagnostic artifacts are reproducible. Happy to extract any subset (just the probe, just the corpus, individual minimal repros) into a form most useful for triage. Fixes in my Obj-C port landed in ~80 lines of code total; I'd expect the Swift equivalents to be similarly small.

For details see:
[swift-transformers-issue.md](https://github.com/user-attachments/files/27557954/swift-transformers-issue.md)