We need a better way to retrieve many documents at same time [LUCENE-7029]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
My use case involve that in some situations I need to get a lot of documents from the index. Most of my documents are small (less than 2KB).
I use API: IndexReader.document(id, visitor), but I noticed it keeps decompressing same block over and over again.
If we have an api that let's me get many documents at once, I could do the same job way faster.
After some experiment I realized I can get 2x+ gains on performance with this simple feature.
Disclaimer: I'm a newbie to lucene project. I truly appreciate hard work by all contributors for this awesome library. I would be more than happy to contribute back and get my hands dirty. Any help and pointers is more than welcomed.
---
Migrated from [LUCENE-7029](https://issues.apache.org/jira/browse/LUCENE-7029) by Jacek Migdal
Attachments: [LUCENE-7029.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7029/LUCENE-7029.patch)
Contributor guide
Research direction
Start with the IndexReader.document(id, visitor) entry point and inspect the attached LUCENE-7029.patch. The work is complete when a bulk document-retrieval API avoids repeatedly decompressing the same block and achieves the reported performance improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100