PanicException: byte index is not a char boundary — count_tokens on multibyte-dense input >~11KB
- Lenguaje dominante
- Rust
- Estrellas
- 52
- Forks
- 5
- Merge medio
- 1 min
- PR fusionados (30 d)
- 2
Descripción
`count_tokens` splits long inputs at byte offsets without checking UTF-8 character boundaries. Any text past roughly 11KB that is dense in 3-byte codepoints raises a pyo3 `PanicException` ("byte index is not a char boundary") instead of returning a count.
Hit in production on two real inputs: long CJK prose, and a vision-model caption that rendered a masked password field as a ~4000-char `•` run.
**Versions:** tokie 0.0.10 / chonkie-core 0.10.2 / chonkie 1.6.8 (latest at time of writing), reproduced via chonkie's public API:
```python
from chonkie import RecursiveChunker
chunker = RecursiveChunker.from_recipe(
"markdown", lang="en",
tokenizer="Qwen/Qwen3-Embedding-8B",
chunk_size=384,
)
chunker.chunk("数据库连接配置说明文档" * 400) # panics
chunker.chunk("•" * 4000) # panics
```
Both raise `pyo3_runtime.PanicException` from `count_tokens`. Note the panic is a `BaseException`, not an `Exception`, so a bare `except Exception` in a pipeline won't catch it — without a specific guard it costs the whole document.
**Expected:** the same counts/boundaries HF `tokenizers` produces. We verified the HF backend counts the same vocab without the byte-slicing issue and produces identical boundaries on benign text (that's our current workaround: catch the panic and retry once on an HF-`tokenizers`-backed chunker). Presumably the fix is snapping split offsets to `char_indices` boundaries before slicing.
Happy to provide more repro detail if useful.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.