NVIDIA / NVIDIA/cuvs

[FEA] Find a formula to convert HNSWLIB build params to CAGRA build params (for HNSW search)

Open
#740 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

We need to do more investigation of this compatibility in params. Thus far, I just don't think we have analyzed the Paretos for enough datasets to be able to say one way or another how whether this conversion even makes sense.

The goal here is to say "I have params M and efConstruction for HNSW, what graph_degree and intermediate_graph_degree can I use to build a CAGRA graph that when converted to HNSW will yield a similar recall/throughput curve".

One thing we can (and should) try is taking a bunch of data for Pareto curves and testing the predictive capabilities by building a simple regression model to predict the CAGRA parameters and expected throughput on the Pareto when given HNSWlib parameters. We have done little exercises like this in past for finding things like optimal code paths for kernel launches given various shapes of data.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by collecting Pareto-curve data across more datasets for HNSWLIB and CAGRA, then evaluate a regression model using HNSW M and efConstruction as inputs. Done means establishing whether the model can predict CAGRA parameters and expected throughput for a similar recall/throughput curve.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.