NVIDIA / NVIDIA/cuvs

[FEA] Adding attributes to vectors for indexing and retrieval.

Open
#1,945 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Is your feature request related to a problem? Please describe.

We want to be able to supply some search conditions on predefined attributes on vectors.
We need to retrieve nearest N vectors (and attributes) after applying these search conditions.

Describe the solution you'd like

When creating the vector index, allow user to supply a small number (8 to 10 will be enough) of attributes of each vector. Each attributes is an 64 bit integer. Later during query for N nearest vectors, allow user to pass in a search condition on these attributes. Easiest conditions could be a few AND-ed comparisons, such as
attribute[0] = XXX AND attribute[3] > YYY. Include all attributes in search result as well.

Describe alternatives you've considered

More general search conditions can be useful, such as allow user to supply a UDF, taking attributes as parameter.

Additional context

MatrixOrigin/MatrixOne is a general purpose database with native support for vector data type. Our query workload is often "mixed", that is, some boolean search condition and retrieve top/nearest N vectors that satisfies these boolean search. Current cuVS allow us to supply a bitmap of vector ids but computing this bitmap is not necessarily cheap -- it forces our query engine to process search conditions first, then asking top N from vector index. Pushing these search conditions down into cuVS is essential for query performance.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points. Start by locating the vector-index creation and nearest-N query APIs, then compare their existing vector-ID bitmap filtering path. Done should include a scoped design for 8–10 int64 attributes, AND-ed comparisons, returning attributes with results, and tests covering filtering and retrieval.

Written by the indexing model from the issue text.

Assessment

Domain
databases, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.