NVIDIA / NVIDIA/cuvs

[BUG] Cagra graph build copies dataset multiple times

Open
#1,456 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Describe the bug
When we supply to cagraIndexBuild (or the equivalent C++ functions) a dataset that resides in device memory (e.g. because it was already copied, e.g. to remove strides as a workaround for #1455, or is the result of other GPU computations like quantization), a new copy of the dataset is created in device memory. This means we need (at least) twice the dataset memory on the device, meaning cagraIndexBuild is likely to fail for bigger datasets.

Steps/Code to reproduce bug

On a X GB GPU, supply to cagraIndexBuild a DLManagedTensor with dataset type kDLCUDA (dataset address on device memory) and size X * 0.6 GB (a bit more than half the ram, e.g. a 13GB dataset on a 24GB GPU). cagraIndexBuild fails with a memory error.

Expected behavior

cagraIndexBuild should provide help/guidance/a way to avoid that:

  • a way of not copying accepting the performance hit could be fine
  • instructions on how to shape the data (pre-stridden/padded data) so we can avoid the copy and take a "fast path"

Any way to avoid the issue and allow more data to be fitted and processed on the GPU

Additional context

Another issue that was recently brought to light is the fact that nn-descent wants to copy the whole dataset into float16.
That is a separate issue above and beyond the strided/padded dataset issue, but it's another area of concern around usage of GPU memory/multiple dataset copies.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the cagraIndexBuild API and its equivalent C++ functions, focusing on handling DLManagedTensor inputs with kDLCUDA device memory and the strided or padded dataset path described alongside issue #1455. Reproduce the failure with a dataset using about 60% of GPU memory, then verify that the build avoids an unnecessary dataset copy or clearly documents the required input shape and trade-offs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.