[Feature] ClickHouse catalog: support vector_similarity data skipping index type
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Version
main branch
### Describe the feature
Add support for the `vector_similarity` data skipping index type in the ClickHouse catalog. This index enables approximate nearest neighbor (ANN) search on vector columns via functions like `L2Distance()`, `cosineDistance()`, and `dotProduct()`.
**Name history**: `annoy` (v22.9, experimental) and `usearch` (v23.8, experimental) coexisted as parallel implementations. They were merged into `vector_similarity` in v24.8 (experimental), promoted to GA in v25.8.
| Version | Date | Name | Status |
|---|---|---|---|
| v22.9 | 2022-08 | `annoy` | experimental |
| v23.8 | 2023-08 | `usearch` | experimental |
| v24.8 | 2024-08 | `vector_similarity` | experimental |
| v25.8 | 2025-08 | `vector_similarity` | GA |
**Current behavior**: Tables with `vector_similarity` (or legacy `annoy`/`usearch`) indexes load without error but index metadata is silently dropped (warning log). Explicit creation attempts throw `IllegalArgumentException`.
**Expected behavior**: Index metadata is preserved on table load, and tables can be created with `vector_similarity` index type via Gravitino API.
### Proposed implementation
1. Add `DATA_SKIPPING_VECTOR_SIMILARITY` to `Index.IndexType` enum
2. Read path: map `vector_similarity`, `annoy`, and `usearch` to `DATA_SKIPPING_VECTOR_SIMILARITY`
3. Write path: generate DDL with required parameters
4. Pass parameters via `Index.properties()`
### Index parameters
Three required positional parameters, plus optional HNSW tuning parameters:
| Parameter | Required | Default | Description |
|---|---|---|---|
| `type` | yes | — | Search method. Currently only `hnsw` |
| `distance_function` | yes | — | `L2Distance`, `cosineDistance`, or `dotProduct` |
| `dimensions` | yes | — | Expected array cardinality (e.g., 384, 768, 1536) |
| `quantization` | no | `bf16` | `f64`, `f32`, `f16`, `bf16`, `i8`, `b1` |
| `hnsw_max_connections_per_layer` | no | 32 | M hyperparameter |
| `hnsw_candidate_list_size_for_construction` | no | 128 | ef_construction hyperparameter |
DDL format:
```sql
INDEX idx vec TYPE vector_similarity('hnsw', 'L2Distance', 384) GRANULARITY 100000000
INDEX idx vec TYPE vector_similarity('hnsw', 'cosineDistance', 768, 'bf16', 32, 128) GRANULARITY 100000000
```
Note: The default GRANULARITY for `vector_similarity` is 100,000,000 (unlike other skip indexes which default to 1). This is because ANN search requires a large number of vectors within a single index block to be effective. Also, `dotProduct` requires `ORDER BY ... DESC`; `L2Distance` and `cosineDistance` require `ORDER BY ... ASC`.
### References
- Annoy introduction: [ClickHouse/ClickHouse#37215](https://github.com/ClickHouse/ClickHouse/pull/37215) (v22.9, 2022-08)
- USearch introduction: [ClickHouse/ClickHouse#53447](https://github.com/ClickHouse/ClickHouse/pull/53447) (v23.8, 2023-08)
- Merge to vector_similarity: [ClickHouse/ClickHouse#63675](https://github.com/ClickHouse/ClickHouse/pull/63675) (v24.8, 2024-08)
- Related: #11912
Contributor guide
Research direction
Start at the ClickHouse catalog's Index.IndexType enum and trace the existing read and write paths for data-skipping indexes. Check how Index.properties() carries parameters; done means vector_similarity, annoy, and usearch metadata survives table loading and the API can create vector_similarity indexes with the required DDL parameters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, java
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100