[FEA] Strongly filtered CAGRA
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
CAGRA has been observed to yield low recall when filtering is enabled, especially when the ratio of filtered-out values is high. This can be related in part to #208 and #472 , but there also may be fundamental reasons for the lower recall.
This feature request tracks the progress and suggestions to enable high-recall strongly filtered CAGRA.
As an experiment, I suggest to try the following tweaks, enabled by a boolean search parameter:
- Disable the maximum search iterations limit to allow longer search
- Replace the hashmap with a dataset-long bitset. It's used to track the visited nodes. By replacing a small hashmap with the bitset we will eliminate hash collisions (thus, false-positives) and prevent CAGRA from early-stopping.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with cpp/include/cuvs/neighbors/cagra.hpp and cpp/src/neighbors/detail/cagra/hashmap.hpp to understand the existing search-iteration limit, search parameters, and visited-node tracking. Compare the proposed parameter-controlled alternatives and establish how high recall under strong filtering would be measured; done requires an agreed design and validation results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100