[BUG] Lucene: preserve released binary-vector indexes while enabling binary storage
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Problem
Released cuvs-lucene binary-quantized codecs wrote Lucene99 flat-vector files because version requests shared one LuceneProvider. Switching the existing codec identity directly to Lucene102 binary storage would make those released segments unreadable by the new implementation.
Proposed change
- Mark newly written segments with a stable NVIDIA-owned flat-layout attribute.
- Write new segments with Lucene102 binary flat storage.
- Read marked segments with the binary reader and unmarked released segments with the legacy Lucene99 flat reader.
- Rewrite legacy segments into the marked layout during merge.
- Fail clearly for unknown or contradictory markers.
Add an index fixture generated by released 26.08 code plus open/search, mixed-segment merge, CPU fallback, and GPU coverage. Preserve the existing codec and SPI identities and new-reader/old-index compatibility.
Related to #2512.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing LuceneProvider and codec/SPI identities, then inspect how released 26.08 indexes are opened and searched. Done means preserving old-index compatibility while covering marked and unmarked segments, mixed-segment merge, CPU fallback, GPU coverage, and clear handling of unknown or contradictory markers.
Written by the indexing model from the issue text.
Assessment
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100