[Feature] ClickHouse catalog: support text/gin/full_text/inverted data skipping index types
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Version
main branch
### Describe the feature
Add support for the full-text search data skipping index type in the ClickHouse catalog. This index has been renamed across ClickHouse versions:
| Version range | Name | Status |
|---|---|---|
| v23.2 ~ v24.4 | `inverted` | experimental |
| v24.5 ~ v25.5 | `full_text` | experimental |
| v25.6 ~ v26.1 | `text` | experimental |
| v26.2+ | `text` | GA |
Note: `gin` appeared as an intermediate name within v25.6 (full_text → gin → text, both renames landed in the same release) but was never the stable name of a published release. The read path should still handle `gin` for compatibility with indexes created via API during that transition.
Each rename is a backward incompatible change: old index names remain loadable but throw an exception when used in searches.
**Current behavior**: Tables with `text`/`gin`/`full_text`/`inverted` indexes load without error but index metadata is silently dropped (warning log). Explicit creation attempts throw `IllegalArgumentException`.
**Expected behavior**: Index metadata is preserved on table load across all name variants, and tables can be created with the appropriate index type via Gravitino API.
### Proposed implementation
1. Add `DATA_SKIPPING_TEXT` to `Index.IndexType` enum (using the latest GA name `text` as the canonical type)
2. Read path: map all four names (`text`, `gin`, `full_text`, `inverted`) to `DATA_SKIPPING_TEXT` for backward compatibility
3. Write path: generate DDL using the name that matches the target ClickHouse server version
4. Pass parameters via `Index.properties()`
### Write path version strategy
The index name changes across versions. The write path should generate DDL compatible with the target ClickHouse server:
| Version range | Name | DDL example |
|---|---|---|
| v23.2 ~ v24.4 | `inverted` | `TYPE inverted(0) GRANULARITY 1` |
| v24.5 ~ v25.5 | `full_text` | `TYPE full_text(0) GRANULARITY 1` |
| v25.6 ~ v26.1 | `text` | `TYPE text(tokenizer = splitByNonAlpha) GRANULARITY 1` |
| v26.2+ | `text` (GA) | `TYPE text(tokenizer = splitByNonAlpha) GRANULARITY 1` |
Options for implementation:
- **Option A**: Always use `text` (latest name). Works on v25.6+, fails on older.
- **Option B**: User-configurable property (e.g., `index_type_name`). Most flexible.
- **Option C**: Auto-detect ClickHouse server version and choose the appropriate name. Best UX.
### Index parameters
Two tokenizer modes, passed via `Index.properties()`:
| Parameter | Required | Default | Description |
|---|---|---|---|
| `tokenizer` | no | `tokens` | `tokens` (splitByNonAlpha) or `ngrams` |
| `ngram_size` | only when tokenizer=ngrams | — | N in ngrams(N), range 2~8 |
Note: The `text` index type (v26.2+ GA) supports additional parameters (preprocessor, postprocessor, positions, dictionary_block_size, etc.) which can be added in a follow-up enhancement.
### References
- Inverted index introduction: [ClickHouse/ClickHouse#38667](https://github.com/ClickHouse/ClickHouse/pull/38667) (v23.2, 2023-01)
- Rename to full_text: [ClickHouse/ClickHouse#62884](https://github.com/ClickHouse/ClickHouse/pull/62884) (v24.5, 2024-04)
- Rename to gin then text (both in v25.6): [ClickHouse/ClickHouse#79024](https://github.com/ClickHouse/ClickHouse/pull/79024), [ClickHouse/ClickHouse#80855](https://github.com/ClickHouse/ClickHouse/pull/80855) (v25.6, 2025-06)
- Related: #11912
Contributor guide
Research direction
Start at the ClickHouse catalog's Index.IndexType handling and trace the read and write paths for data-skipping indexes. Review how Index.properties() is passed and how server-version compatibility is determined. Done means all four legacy names load as preserved metadata and creation generates compatible DDL with tokenizer parameters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100