ClickHouse / ClickHouse/ClickHouse

Text index: Add a StringZilla-based tokenizer

Open
#111,696 3 comments 2 reactions 1 assignee Claimed by @albin-george-kurian View on GitHub
comp-text-index easy task feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

ClickHouse's [text index](https://clickhouse.com/docs/engines/table-engines/mergetree-family/textindexes) provides the `asciiCJK` tokenizer (first added [here](https://github.com/ClickHouse/ClickHouse/pull/99357), it was later renamed). It uses ICU to split Chinese, Japanese and Korean inputs into tokens.

The recent [StringZilla 5.0 release](https://x.com/ashvardanian/status/2079592324099625250) added equivalent functionality (maybe even a drop-in replacement) to tokenize inputs, see [the docs here](https://github.com/ashvardanian/StringZilla/blob/main/include/stringzilla/README.md). I'm pretty sure that @ashvardanian (main author of StringZilla) went great length to make the tokenization in StringZilla faster than in ICU.

We should test that out. Basically:
- bump the existing StringZilla submodule in ClickHouse to 5.0
- introduce a new tokenizer `asciiCJK_v2` based on `asciiCJK` but using Stringzilla's UTF8 segmentation (in case of success, the latter will be deprecated)
- do benchmarking ... lots of benchmarking to prove the new tokenizer has a tangible performance benefit

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.