[BUG] Iterative CAGRA graph build fails while benchmarking with override_kv=dataset_memory_type:"mmap"
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Iterative CAGRA graph build fails with an exception
CUDA call='cudaFreeAsync(ptr, stream)' at file=/workspace/repos/cuvs/cpp/src/neighbors/detail/cagra/compute_distance.hpp line=248 failed with misaligned address
override_kv=dataset_memory_type:"device" and override_kv=dataset_memory_type:"host" work
** To reproduce **
Run
CUVS_CAGRA_ANN_BENCH --build --force --data_prefix=/workspace/datasets/ --benchmark_out_format=csv --benchmark_out=res_build.csv --benchmark_counters_tabular=true --override_kv=dataset_memory_type:"mmap" /workspace/repos/vdb_bench/laion_1M_cagra_iterative.json
** laion_1M_cagra_iterative.json **
{
"dataset": {
"name": "laion_1M",
"base_file": "laion_1M/base.1M.fbin",
"subset_size": 1000000,
"query_file": "laion_1M/queries.fbin",
"groundtruth_neighbors_file": "laion_1M/groundtruth.1M.neighbors.ibin",
"distance": "euclidean"
},
"search_basic_param": {
"batch_size": 10000,
"k": 10
},
"index": [
{
"name": "cuvs_cagra_iterative",
"algo": "cuvs_cagra",
"build_param": {
"graph_degree": 64,
"intermediate_graph_degree": 128,
"search_width": 1
},
"file": "laion_1M/cagra/q_coarse.ibin",
"search_params": [
{"itopk": 64, "search_width": 2, "max_iterations": 8},
{"itopk": 128, "search_width": 1, "max_iterations": 12},
{"itopk": 128, "search_width": 1, "max_iterations": 16},
{"itopk": 256, "search_width": 1, "max_iterations": 16},
{"itopk": 512, "search_width": 2, "max_iterations": 10},
{"itopk": 256, "search_width": 2, "max_iterations": 12},
{"itopk": 32, "search_width": 1, "max_iterations": 32},
{"itopk": 32, "search_width": 1, "max_iterations": 64},
{"itopk": 192, "search_width": 4, "max_iterations": 12},
{"itopk": 256, "search_width": 4, "max_iterations": 12},
{"itopk": 256, "search_width": 16, "max_iterations": 10},
{"itopk": 512, "search_width": 16, "max_iterations": 32}
]
}
]
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with CUVS_CAGRA_ANN_BENCH and laion_1M_cagra_iterative.json using dataset_memory_type:"mmap". Inspect cpp/src/neighbors/detail/cagra/compute_distance.hpp around line 248 and compare the mmap path with the working device and host paths; done means iterative graph building completes without the misaligned-address exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100