apache / apache/lucene

We need a better way to retrieve many documents at same time [LUCENE-7029]

Open
#8,086 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:core/codecs module:core/index type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

My use case involve that in some situations I need to get a lot of documents from the index. Most of my documents are small (less than 2KB).

I use API: IndexReader.document(id, visitor), but I noticed it keeps decompressing same block over and over again.

If we have an api that let's me get many documents at once, I could do the same job way faster.

After some experiment I realized I can get 2x+ gains on performance with this simple feature.

Disclaimer: I'm a newbie to lucene project. I truly appreciate hard work by all contributors for this awesome library. I would be more than happy to contribute back and get my hands dirty. Any help and pointers is more than welcomed.

---
Migrated from [LUCENE-7029](https://issues.apache.org/jira/browse/LUCENE-7029) by Jacek Migdal
Attachments: [LUCENE-7029.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7029/LUCENE-7029.patch)

Contributor guide

Open the contributing guide

Research direction

Start with the IndexReader.document(id, visitor) entry point and inspect the attached LUCENE-7029.patch. The work is complete when a bulk document-retrieval API avoids repeatedly decompressing the same block and achieves the reported performance improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.