Lucene: CAGRA IVF-PQ graph build fails when `refinement_rate=2` — IVF list capacity too small (128 vs 467)

Summary

Building a CAGRA graph with the IVF-PQ build algorithm fails when refinement_rate=2. The same build succeeds with refinement_rate=1.

The failure happens inside fill_refinement_index: cuVS tries to store ~467 records but the IVF list is capped at 128 slots, then GPU allocation fails.

Expected

refinement_rate=2 should build a valid CAGRA graph (possibly using more memory/time than refinement_rate=1).

Actual

Build crashes during graph construction with:

RAFT failure at ivf_list.cuh line=52:
ivf::list: failed to allocate a big enough list to hold all data
(requested size: 467 records, selected capacity: 128 records)
cudaErrorMemoryAllocation

Stack includes: fill_refinement_index → build_knn_graph (IVF_PQ) → cuvsCagraBuild

Environment

• GPU: NVIDIA L4 (23 GB)
• cuVS Java: 26.06.0
• Dataset: 1M vectors, 768 dimensions (float32)
• Build algo: CAGRA graph build = IVF_PQ
• Key params: n_lists=1024, n_probes=10, graph_degree=32, intermediate_graph_degree=64

Minimal reproducer

Only change refinement_rate from 1 to 2. Everything else stays the same.

Option A — one JSON config + Maven (simplest if you use vectorsearch-benchmarks)

1. Use config with refinement_rate=2:

{
  "datasetFile": "/path/to/base.1M.fbin",
  "queryFile": "/path/to/queries.fbin",
  "groundTruthFile": "/path/to/groundtruth.1M.neighbors.ibin",
  "vectorDimension": 768,
  "numDocs": 1000000,
  "numQueriesToRun": 100,
  "numWarmUpQueries": 0,
  "flushFreq": 10000000,
  "topK": 100,
  "vectorColName": "vector",
  "algoToRun": "CAGRA_HNSW",
  "indexDirPath": "cuvsIndex-test",
  "cagraGraphBuildAlgo": "IVF_PQ",
  "cuvsWriterThreads": 4,
  "numIndexThreads": 4,
  "cagraGraphDegree": 32,
  "cagraIntermediateGraphDegree": 64,
  "cagraHnswLayers": 1,
  "cuVSIvfPqIndexParamsNLists": 1024,
  "cuVSIvfPqSearchParamsNProbes": 10,
  "cuVSIvfPqParamsRefinementRate": 2
}

2. Run:

export LD_LIBRARY_PATH=/path/to/cuvs/cpp/build/c:/usr/local/cuda/lib64:$LD_LIBRARY_PATH

mvn exec:java \
  -Dexec.mainClass=com.searchscale.lucene.cuvs.benchmarks.LuceneCuvsBenchmarks \
  -Dexec.args="refine2-config.json TEST /tmp/results" \
  -Dexec.jvmArgs="--add-modules=jdk.incubator.vector --enable-native-access=ALL-UNNAMED"

3. Works if you set "cuVSIvfPqParamsRefinementRate": 1
4. Fails if you set "cuVSIvfPqParamsRefinementRate": 2 (after indexing ~1M docs, at commit/merge)

Option B — cuVS C++ level (conceptual)

Same IVF-PQ params passed to CAGRA build:

graph_build_params::ivf_pq_params params;
params.build_params.n_lists = 1024;
params.search_params.n_probes = 10;
params.refinement_rate = 2.0f;   // fails
// params.refinement_rate = 1.0f; // ok

// cuvs::neighbors::cagra::build(res, index_params, dataset);
// index_params.graph_build_params = params;

Why this looks like a cuVS bug

• refinement_rate=2 doubles the refinement candidate count (gpu_top_k = graph_degree * refinement_rate).
• The code path needs an IVF list with 467 entries, but allocation stops at 128 (power-of-two cap in ivf_list.cuh).
• Failure happens even with 4 writer threads (not just under heavy parallel merge load).
• GPU has plenty of free memory (~23 GB L4, nearly idle) — error is about list sizing, not total VRAM exhaustion.

Workaround

Use refinement_rate=1 until list sizing supports higher refinement rates for this dataset size.

Start with the IVF-PQ CAGRA path around fill_refinement_index, build_knn_graph, and ivf_list.cuh, using the reported 467-versus-128 capacity failure as the guide. Reproduce with the provided Maven configuration, comparing refinement_rate=2 against 1. Done means the refinement-rate-2 configuration builds a valid graph without the IVF list allocation failure.