dolthub / dolthub/dolt

Support vector index on multiple columns

Open
#8,661 0 comments 0 reactions 1 assignee Claimed by @zachmu View on GitHub
enhancement indexes vectors
Dominant language
Go
Stars
24.4k
Forks
873
Avg merge
1d 5h
Merged PRs (30d)
108

Description

Currently our vector index workflow is based on MariaDB, which defines vector indexes on a single column that contains the entire vector. We currently support vector indexes on JSON columns and plan to add support for a dedicated `VECTOR` type.

However, some vector datasets on HuggingFace take a different approach and store each dimension of the vector in a different float column. Importing these datasets into Dolt in a way that makes them indexible requires extra steps to convert the representation of the data.

But there's no reason why we shouldn't be able to support vector indexes on multiple float columns. For example:

```
CREATE TABLE test(pk int primary key, v0 float, v1 float, v2 float, v3 float);
CREATE INDEX vidx on TEST(v0, v1, v2, v3);
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.