NVIDIA / NVIDIA/cuvs

[FEA] Add filtering option to benchmarks

Open
#479 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Is your feature request related to a problem? Please describe.
As cuVS algorithms get more pre-filtering support, we need the be able to benchmark this functionality and compare the algorithms

Describe the solution you'd like

  • Add a search-only option to the benchmark executable --filter_ratio x, where x is a float between 0 and 1 means the proportion of the records in the index passing the filter; by default it's 1 meaning the legacy behavior (no filtering).
  • Modify the dataset class: add an extra bitset field of the same size as the dataset itself; allow generating it (or loading from file? when generating, maybe also expose the random seed parameter?).
    • Note 1: this way, the filter is set up once per whole benchmark; this ensures low overheads and a fair comparison.
    • Note 2: we cannot use raft's bitset here, because the common benchmark headers don't depend on raft.
  • Pass the bitset filter to the algorithms. I think, the easiest way would be to add a new api function set_filter(bitset ptr) similar to set_search_parameters. It would be called once per benchmark loop and only if the filter ratio is lower than 1.
  • Adapt the ground truth calculation. I think, the easiest way here would be to replace the total_count in the calculation of the recall with the number of non-filtered items. This will have noise (when the count is low) and will not be entirely correct (the recall is averaged across threads/loops), but it's better than nothing.
    • Note: there's a way to enhance the quality at somewhat low effort when k is smaller than the available max_k in the ground truth file; that is to consider not first k values in the ground truth, but first k non-filtered values in the ground truth.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the benchmark executable, the dataset class, and cpp/bench/ann/src/common/benchmark.hpp, especially the recall calculation around total_count. Trace how search parameters and ground truth are passed through the benchmark loop; done means --filter_ratio, one reusable filter bitset, algorithm filter propagation, and filtered recall behavior are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.