apache / apache/gravitino

[Feature] ClickHouse catalog: support vector_similarity data skipping index type

Open
#11,935 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Version

main branch

### Describe the feature

Add support for the `vector_similarity` data skipping index type in the ClickHouse catalog. This index enables approximate nearest neighbor (ANN) search on vector columns via functions like `L2Distance()`, `cosineDistance()`, and `dotProduct()`.

**Name history**: `annoy` (v22.9, experimental) and `usearch` (v23.8, experimental) coexisted as parallel implementations. They were merged into `vector_similarity` in v24.8 (experimental), promoted to GA in v25.8.

| Version | Date | Name | Status |
|---|---|---|---|
| v22.9 | 2022-08 | `annoy` | experimental |
| v23.8 | 2023-08 | `usearch` | experimental |
| v24.8 | 2024-08 | `vector_similarity` | experimental |
| v25.8 | 2025-08 | `vector_similarity` | GA |

**Current behavior**: Tables with `vector_similarity` (or legacy `annoy`/`usearch`) indexes load without error but index metadata is silently dropped (warning log). Explicit creation attempts throw `IllegalArgumentException`.

**Expected behavior**: Index metadata is preserved on table load, and tables can be created with `vector_similarity` index type via Gravitino API.

### Proposed implementation

1. Add `DATA_SKIPPING_VECTOR_SIMILARITY` to `Index.IndexType` enum
2. Read path: map `vector_similarity`, `annoy`, and `usearch` to `DATA_SKIPPING_VECTOR_SIMILARITY`
3. Write path: generate DDL with required parameters
4. Pass parameters via `Index.properties()`

### Index parameters

Three required positional parameters, plus optional HNSW tuning parameters:

| Parameter | Required | Default | Description |
|---|---|---|---|
| `type` | yes | — | Search method. Currently only `hnsw` |
| `distance_function` | yes | — | `L2Distance`, `cosineDistance`, or `dotProduct` |
| `dimensions` | yes | — | Expected array cardinality (e.g., 384, 768, 1536) |
| `quantization` | no | `bf16` | `f64`, `f32`, `f16`, `bf16`, `i8`, `b1` |
| `hnsw_max_connections_per_layer` | no | 32 | M hyperparameter |
| `hnsw_candidate_list_size_for_construction` | no | 128 | ef_construction hyperparameter |

DDL format:
```sql
INDEX idx vec TYPE vector_similarity('hnsw', 'L2Distance', 384) GRANULARITY 100000000
INDEX idx vec TYPE vector_similarity('hnsw', 'cosineDistance', 768, 'bf16', 32, 128) GRANULARITY 100000000
```

Note: The default GRANULARITY for `vector_similarity` is 100,000,000 (unlike other skip indexes which default to 1). This is because ANN search requires a large number of vectors within a single index block to be effective. Also, `dotProduct` requires `ORDER BY ... DESC`; `L2Distance` and `cosineDistance` require `ORDER BY ... ASC`.

### References

- Annoy introduction: [ClickHouse/ClickHouse#37215](https://github.com/ClickHouse/ClickHouse/pull/37215) (v22.9, 2022-08)
- USearch introduction: [ClickHouse/ClickHouse#53447](https://github.com/ClickHouse/ClickHouse/pull/53447) (v23.8, 2023-08)
- Merge to vector_similarity: [ClickHouse/ClickHouse#63675](https://github.com/ClickHouse/ClickHouse/pull/63675) (v24.8, 2024-08)
- Related: #11912

Contributor guide

Open the contributing guide

Research direction

Start at the ClickHouse catalog's Index.IndexType enum and trace the existing read and write paths for data-skipping indexes. Check how Index.properties() carries parameters; done means vector_similarity, annoy, and usearch metadata survives table loading and the API can create vector_similarity indexes with the required DDL parameters.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, java
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.