[Bug] cuVS CAGRA concurrent search performance gap between Python and C++ API

Summary

Simulating online search (concurrency = n, batch_size = 1) with cagra search python API.

Impact

Python users have no path to fan out concurrent searches; the safe alternative is multi-process, but each process duplicates the index+dataset on GPU and OOMs quickly. Result: Python tops out around ~600 QPS at bs=1, while cuvs-bench C++ achieves ~30K QPS at 90% recall on the same hardware (~50x gap).

Environment

Setup

Tried (but failed)

Calling cuvs.neighbors.cagra.search from multiple Python threads, each with its own cuvs.common.Resources(stream=stream.ptr), consistently segfaults inside pylibraft.common.device_ndarray.empty during output-buffer allocation. This is the natural Python equivalent of the std::async pattern in examples/cpp/src/cagra_persistent_example.cu, so concurrent submission to the persistent CAGRA kernel is unreachable from Python.

Steps to reproduce

import threading
import cupy as cp
import numpy as np
from cuvs.common import Resources
from cuvs.neighbors import cagra

rng = np.random.default_rng(0)
N, D, Q, K = 100_000, 2048, 8800, 10
base = cp.asarray(rng.random((N, D), dtype=np.float32))
queries = cp.asarray(rng.random((Q, D), dtype=np.float32))

idx = cagra.build(
    cagra.IndexParams(graph_degree=32, intermediate_graph_degree=64, build_algo="nn_descent"),
    base,
)
sp = cagra.SearchParams(itopk_size=64)
_ = cagra.search(sp, idx, queries, K)
cp.cuda.Device().synchronize()

def worker(a, b):
    stream = cp.cuda.Stream(non_blocking=True)
    res = Resources(stream=stream.ptr)
    cagra.search(sp, idx, queries[a:b], K, resources=res)
    res.sync()

threads = [threading.Thread(target=worker, args=(0, Q // 2)),
           threading.Thread(target=worker, args=(Q // 2, Q))]
for t in threads: t.start()
for t in threads: t.join()

Repo -

cuvs-search-bench

Start with the Python cagra.search path and the pylibraft.common.device_ndarray.empty failure described in the reproducer. Compare it with examples/cpp/src/cagra_persistent_example.cu and the cuvs-bench setup, then investigate safe concurrent submission and output allocation. Done means concurrent Python searches no longer segfault and the performance gap is characterized or resolved.