NVIDIA / NVIDIA/cuvs

Lucene: Dataset might get flushed twice when using CAGRA build

Open
#2,457 0 comments 0 reactions 1 assignee View on GitHub

@imotov is already working on this.

Since Aug 17, 2026.

Lucene
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

I ran some experiments with vectorsearch-benchmarks and enabled logging with enableIndexWriterInfoStream. I consistently see about twice the flush size for each segment when using CAGRA_SEARCH instead of CAGRA_HNSW. It looks like the CAGRA index and the dataset is flushed to disk using CagraIndex.serialize() (CuVS2510GPUVectorsWriter.java#L257) which sets include_dataset by default (CagraIndexImpl.java#L369. FlatVectorsWriter writes the dataset to the .vec file regardless. CAGRA_HNSW flushes the adjacency list only (LuceneAcceleratedHNSWScalarQuantizedVectorsWriter.java#L206.

We might want to add a cuvs Java interface for CagraIndex.serialize(OutputStream os, boolean includeDataset) and pass serialize(os, /*includeDataset=*/false) to cuvs-lucene in CuVS2510GPUVectorsWriter.writeCagraIndex().

CC @cjnolet

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.