ClickHouse / ClickHouse/ClickHouse
Text index: Add a StringZilla-based tokenizer
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
ClickHouse's [text index](https://clickhouse.com/docs/engines/table-engines/mergetree-family/textindexes) provides the `asciiCJK` tokenizer (first added [here](https://github.com/ClickHouse/ClickHouse/pull/99357), it was later renamed). It uses ICU to split Chinese, Japanese and Korean inputs into tokens.
The recent [StringZilla 5.0 release](https://x.com/ashvardanian/status/2079592324099625250) added equivalent functionality (maybe even a drop-in replacement) to tokenize inputs, see [the docs here](https://github.com/ashvardanian/StringZilla/blob/main/include/stringzilla/README.md). I'm pretty sure that @ashvardanian (main author of StringZilla) went great length to make the tokenization in StringZilla faster than in ICU.
We should test that out. Basically:
- bump the existing StringZilla submodule in ClickHouse to 5.0
- introduce a new tokenizer `asciiCJK_v2` based on `asciiCJK` but using Stringzilla's UTF8 segmentation (in case of success, the latter will be deprecated)
- do benchmarking ... lots of benchmarking to prove the new tokenizer has a tangible performance benefit
Contributor guide
Assessment
This issue has not been assessed yet.