ClickHouse / ClickHouse/ClickHouse

Add SCANN Vector Index Support

Open
#103,664 3 comments 3 reactions 0 assignees View on GitHub
comp-vector-index external feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

MyScaleDB

### Use case

ClickHouse currently supports vector similarity search via HNSW (through USearch). However, HNSW has known limitations in high-recall scenarios: as ef_search increases to achieve higher recall, QPS drops significantly due to the graph traversal nature of the algorithm.

We believe there is value in offering an alternative vector index that can deliver higher QPS at high recall levels, particularly for workloads where data fits in memory and recall requirements are strict.

### Describe the solution you'd like

We would like to ask whether the ClickHouse community would be open to accepting a contribution of SCANN (Scalable Nearest Neighbors) as an additional vector index.

SCANN offers meaningful advantages over HNSW in high-recall scenarios:
- Uses anisotropic quantization and IVF-based search, achieving significantly higher QPS at the same recall level
- A strong complement to the existing USearch-based HNSW index

We are the team behind MyScaleDB (https://github.com/OriginHubAI/MyScaleDB), a ClickHouse-based database with extended vector search capabilities. We already have a working SCANN integration in our open-source library search-index-oss (https://github.com/OriginHubAI/search-index-oss), which gives us deep familiarity with SCANN's build/train/search lifecycle, parameter tuning, and serialization.

We would like to know if ClickHouse has any plans to add SCANN vector index support. We would love to collaborate with the team and contribute this work.

### Describe alternatives you've considered

We previously considered contributing our entire vector indexing framework (search-index-oss) as an abstraction layer on top of ClickHouse's existing interfaces. Based on feedback from the ClickHouse team, we understand this is not the preferred direction, and we fully agree that contributing SCANN directly using ClickHouse's native interfaces is a cleaner approach.

### Additional context

Our long-term goal is to gradually contribute our vector search research and engineering — built through years of work on MyScaleDB — back to the ClickHouse community.

We are also closely following the Vector Similarity Index 2.0 discussion at https://github.com/ClickHouse/ClickHouse/issues/102146 and are keen to be involved in that direction as it evolves.

References:
- search-index-oss: https://github.com/OriginHubAI/search-index-oss
- MyScaleDB: https://github.com/OriginHubAI/MyScaleDB

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.