NVIDIA / NVIDIA/cuvs

GPU search hardcodes query batch size 1 — no multi-query batching

Open
#2,464 2 comments 0 reactions 1 assignee View on GitHub

@imotov is already working on this.

Since Aug 17, 2026.

Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Summary

cuvs-lucene always passes a 1 × dim query matrix to cuVS. CagraSearchParams.maxQueries is never set. cuVS supports batched search natively; the Lucene layer does not expose it.

Reproducer

From a cuvs-lucene checkout:

rg 'deviceBuilder\([^,]+, 1,' src/main/java/com/nvidia/cuvs/lucene/

Both GPU search paths hardcode batch size 1:

  • GPUKnnFloatVectorQuery.java (~line 195) — multi-segment path
  • CuVS2510GPUVectorsReader.java (~line 458) — per-segment fallback

Neither calls CagraSearchParams.Builder.withMaxQueries(n) for n > 1.

Runtime check: run any GPUKnnFloatVectorQuery search; one cuVS cagraSearch call is issued per Lucene query. MultiPartitionCagraSearch batches segments, not query vectors — still one query per call.

Expected vs actual

  • Expected: option to search N query vectors in one GPU call (as cuVS Java API allows via N-row CuVSMatrix + maxQueries).
  • Actual: one query vector per search; throughput scaling in benchmarks uses queryThreads (N concurrent batch-1 searches).

Workaround

Use multiple search threads (queryThreads > 1). Not equivalent to in-kernel query batching.

Environment

cuvs-lucene 25.10+, CuVS2510GPUSearchCodec, GPUKnnFloatVectorQuery.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.