Allow composite readers to have more than 2B documents [LUCENE-8321]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I would like to start discussing removing the limit of \~2B documents that we have for indices, while still enforcing it at the segment level for practical reasons.
Postings, stored fields, and all other codec APIs would keep working on integers to represent doc ids. Only top-level doc ids and numbers of documents would need to move to a long. I say "only" because we now mostly consume indices per-segment, but there is still a number of places where we identify documents by their top-level doc ID like `IndexReader#document`, top-docs collectors, etc.
---
Migrated from [LUCENE-8321](https://issues.apache.org/jira/browse/LUCENE-8321) by Adrien Grand (@jpountz), updated Feb 14 2020
Contributor guide
Research direction
The issue names IndexReader#document, top-docs collectors, and codec APIs as affected areas; start by tracing how top-level document IDs and document counts move across composite readers and segments. Done means top-level counts and IDs support long values while segment-level document IDs remain integers, with the affected behavior verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100