[QST] How to choose parameters for CAGRA?
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
I want to select a good choice of parameters that would give me high recall (>95%) for two large datasets that I have. One of them has 120M rows and the other dataset has 50M rows. I have access to an NVIDIA A100 GPU with 80G VRAM. I want to compute 50 neighbors for every data point and so far the only change I have made is use itopk_size=128 when searching. Since datasets are too large I am not able to perform a brute force search and calculate recall. I was wondering if there are any guidelines on choosing parameters.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing CAGRA's search-parameter guidance, focusing on itopk_size and the requested 50-neighbor search. Use the 120M- and 50M-row datasets with the available A100 as the evaluation context. Done means documenting practical parameter-selection guidance for reaching the requested recall when brute-force recall measurement is unavailable.
Written by the indexing model from the issue text.
Assessment
- Domain
- search
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100