apache / apache/lucene

Allow composite readers to have more than 2B documents [LUCENE-8321]

Open
#9,368 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I would like to start discussing removing the limit of \~2B documents that we have for indices, while still enforcing it at the segment level for practical reasons.

Postings, stored fields, and all other codec APIs would keep working on integers to represent doc ids. Only top-level doc ids and numbers of documents would need to move to a long. I say "only" because we now mostly consume indices per-segment, but there is still a number of places where we identify documents by their top-level doc ID like `IndexReader#document`, top-docs collectors, etc.

---
Migrated from [LUCENE-8321](https://issues.apache.org/jira/browse/LUCENE-8321) by Adrien Grand (@jpountz), updated Feb 14 2020

Contributor guide

Open the contributing guide

Research direction

The issue names IndexReader#document, top-docs collectors, and codec APIs as affected areas; start by tracing how top-level document IDs and document counts move across composite readers and segments. Done means top-level counts and IDs support long values while segment-level document IDs remain integers, with the affected behavior verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.