[FEA] Adding attributes to vectors for indexing and retrieval.
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Is your feature request related to a problem? Please describe.
We want to be able to supply some search conditions on predefined attributes on vectors.
We need to retrieve nearest N vectors (and attributes) after applying these search conditions.
Describe the solution you'd like
When creating the vector index, allow user to supply a small number (8 to 10 will be enough) of attributes of each vector. Each attributes is an 64 bit integer. Later during query for N nearest vectors, allow user to pass in a search condition on these attributes. Easiest conditions could be a few AND-ed comparisons, such as
attribute[0] = XXX AND attribute[3] > YYY. Include all attributes in search result as well.
Describe alternatives you've considered
More general search conditions can be useful, such as allow user to supply a UDF, taking attributes as parameter.
Additional context
MatrixOrigin/MatrixOne is a general purpose database with native support for vector data type. Our query workload is often "mixed", that is, some boolean search condition and retrieve top/nearest N vectors that satisfies these boolean search. Current cuVS allow us to supply a bitmap of vector ids but computing this bitmap is not necessarily cheap -- it forces our query engine to process search conditions first, then asking top N from vector index. Pushing these search conditions down into cuVS is essential for query performance.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by locating the vector-index creation and nearest-N query APIs, then compare their existing vector-ID bitmap filtering path. Done should include a scoped design for 8–10 int64 attributes, AND-ed comparisons, returning attributes with results, and tests covering filtering and retrieval.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100