[FEA] Find a formula to convert HNSWLIB build params to CAGRA build params (for HNSW search)
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
We need to do more investigation of this compatibility in params. Thus far, I just don't think we have analyzed the Paretos for enough datasets to be able to say one way or another how whether this conversion even makes sense.
The goal here is to say "I have params M and efConstruction for HNSW, what graph_degree and intermediate_graph_degree can I use to build a CAGRA graph that when converted to HNSW will yield a similar recall/throughput curve".
One thing we can (and should) try is taking a bunch of data for Pareto curves and testing the predictive capabilities by building a simple regression model to predict the CAGRA parameters and expected throughput on the Pareto when given HNSWlib parameters. We have done little exercises like this in past for finding things like optimal code paths for kernel launches given various shapes of data.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by collecting Pareto-curve data across more datasets for HNSWLIB and CAGRA, then evaluate a regression model using HNSW M and efConstruction as inputs. Done means establishing whether the model can predict CAGRA parameters and expected throughput for a similar recall/throughput curve.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100