[FEA] Add filtering option to benchmarks
Open
Nobody has claimed this yet.
feature request
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Is your feature request related to a problem? Please describe.
As cuVS algorithms get more pre-filtering support, we need the be able to benchmark this functionality and compare the algorithms
Describe the solution you'd like
- Add a search-only option to the benchmark executable
--filter_ratio x, wherexis a float between 0 and 1 means the proportion of the records in the index passing the filter; by default it's 1 meaning the legacy behavior (no filtering). - Modify the dataset class: add an extra bitset field of the same size as the dataset itself; allow generating it (or loading from file? when generating, maybe also expose the random seed parameter?).
- Note 1: this way, the filter is set up once per whole benchmark; this ensures low overheads and a fair comparison.
- Note 2: we cannot use raft's bitset here, because the common benchmark headers don't depend on raft.
- Pass the bitset filter to the algorithms. I think, the easiest way would be to add a new api function
set_filter(bitset ptr)similar toset_search_parameters. It would be called once per benchmark loop and only if the filter ratio is lower than1. - Adapt the ground truth calculation. I think, the easiest way here would be to replace the total_count in the calculation of the recall with the number of non-filtered items. This will have noise (when the count is low) and will not be entirely correct (the recall is averaged across threads/loops), but it's better than nothing.
- Note: there's a way to enhance the quality at somewhat low effort when
kis smaller than the availablemax_kin the ground truth file; that is to consider not firstkvalues in the ground truth, but firstknon-filtered values in the ground truth.
- Note: there's a way to enhance the quality at somewhat low effort when
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the benchmark executable, the dataset class, and cpp/bench/ann/src/common/benchmark.hpp, especially the recall calculation around total_count. Trace how search parameters and ground truth are passed through the benchmark loop; done means --filter_ratio, one reusable filter bitset, algorithm filter propagation, and filtered recall behavior are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100