pingcap / pingcap/tidb

HNSW Index Configuration

Open
#60,429 0 comments 2 reactions 0 assignees View on GitHub
type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

Ability to configure the HNSW Vector Index parameters beyond the distance function.
I see SQL such as:

`alter table t add vector index idx((VEC_COSINE_DISTANCE(b))) USING HNSW
`
But no way to specific parameters of the index like efConstruction.

In TiFlash I see in the code file:
https://github.com/pingcap/tiflash/blob/master/dbms/src/Storages/DeltaMerge/Index/VectorIndex/Reader.cpp
code like:

```
vi->index = USearchImplType::make(
unum::usearch::metric_punned_t( //
file_props.dimensions(),
getUSearchMetricKind(metric)),
unum::usearch::index_dense_config_t(
unum::usearch::default_connectivity(),
unum::usearch::default_expansion_add(),
16 /* default is 64 */));

```
which appears to be setting these to some default values.
In the USearch code these are used like:

```
index_dense_config_t config(args.connectivity, args.expansion_add, args.expansion_search);
...
std::printf("- Index: \n");
std::printf("-- Connectivity: %zu\n", config.connectivity);
std::printf("-- Expansion @ Add: %zu\n", config.expansion_add);
std::printf("-- Expansion @ Search: %zu\n", config.expansion_search);

```
**Describe the feature you'd like:**

It looks like in the design document these were meant to be passed something like:

```
VECTOR INDEX idx_name USING HNSW ((VEC_COSINE_DISTANCE(data))) [WITH OPTION "m=16, ef_construction=64"]
```

https://github.com/pingcap/tidb/blob/master/docs/design/2024-07-12-support-vector-index.md?plain=1

so perhaps this design just needs to be fully realized.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.