[BUG] Filtered CAGRA search returns incorrect recall due to file-backed mmap filter bitset causing a cascade of concurrent page faults
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Describe the bug
Filtered CAGRA search produces recall ≈ pass_rate (e.g., 0.05 for a 5% pass rate filter). The bug is in cuvs_cagra::get_preference() in cpp/bench/ann/src/cuvs/cuvs_cagra_wrapper.h, which returns MemoryType::kHostMmap for dataset_memory_type. The benchmark framework uses this same memory type to provide the filter bitset:
// cpp/bench/ann/src/common/benchmark.hpp (line 266)
a->set_search_param(sp, dataset->filter_bitset(current_algo_props->dataset_memory_type));
kHostMmap resolves to data_mmap(copy_in_memory=false) — a file-backed MAP_PRIVATE mmap with pages not pre-faulted in the CPU page table. GPU kernels access host memory via ATS (Address Translation Services over NVLink-C2C). When a page is not yet in the CPU page table, the GPU access triggers a CPU page fault to bring it in.
During CAGRA graph traversal, thousands of GPU threads simultaneously read random positions across the filter bitset. For a 100M vector dataset, the filter bitset is ~12.5 MB (~3,000 pages). These concurrent random accesses across unmapped pages generate a thundering herd of ATS page faults, flooding the CPU fault handler and producing incorrect filter results.
Steps/Code to reproduce bug
Run filtered CAGRA search on deep-100M with a 5% pass rate filter on GH200:
./cuvs_bench --search --data_prefix=/work \
--benchmark_min_warmup_time=5.0 \
--mode=throughput --threads=32 \
cagra_100M_filtered_5pct.json
Expected recall: ≥ 0.90.
Observed recall: ≈ 0.05 (equal to filter pass rate).
Expected behavior
Filtered CAGRA recall should match filtered IVF-PQ/Flat recall for the same dataset and filter. Changing the memory type to kHost (same as IVF-PQ/Flat, pre-faults pages) or kDevice (explicit copy to GPU memory) fixes the issue.
Environment details (please complete the following information):
- Hardware: NVIDIA GH200 Grace Hopper Superchip (ATS enabled)
- Dataset: deep-100M, 96-dim float32, inner product, 5% filter pass rate
- Confirmed affected: NVIDIA/cuvs main branches as of 2026-07-06
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in cpp/bench/ann/src/cuvs/cuvs_cagra_wrapper.h at cuvs_cagra::get_preference(), then inspect filter_bitset handling in cpp/bench/ann/src/common/benchmark.hpp around line 266. Run the provided cuvs_bench command on a GH200 with the deep-100M 5% filter and compare CAGRA recall with IVF-PQ/Flat; done means filtered CAGRA recall is at least 0.90 and no longer tracks the pass rate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100