apache / apache/lucene

IndexSearcher.doc(int docID, SetfieldsToLoad) is slower in Lucene 4.9 when compared to Lucene 2.9 [LUCENE-6322]

Open
#7,383 4 comments 0 reactions 0 assignees View on GitHub
affects-version:4.9 legacy-jira-fix-version:4.10.5 legacy-jira-priority:Major module:core/codecs type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We use IndexSearcher.doc(int docID, SetfieldsToLoad) method to get the document with selected stored fields. If we did not mention few stored fields which have data more than 500KB, this call is slower in Lucene 4.9 when compared to Lucene 2.9.

I debugged the above method with Lucene 4.9 and found that CompressingStoredFieldsReader#visitDocument(int docID, StoredFieldVisitor visitor) is spending more time while loading file content and decompressing in chunks of 16kb, even to skip the fields. It is noticeable degrade if the document's field size is more than 1MB, and we call this method in loop for more than 1000 such documents.

In case of Lucene 2.9, there was no compression, and if we want to skip the field, it just does file seek to set the next pointer to read the stored field. For example see Lucene3xStoredFieldsReader#skipField() method how it works for skipping a field in Lucene 2.9 which is VERY faster compared to Lucene 4.9.

We should have something in CompressingStoredFieldsReader to know the field’s compressed length in file and just do the file seek to set the next pointer instead of loading content from file and decompress that in 16KB chunks to just skip the field from the file.

---
Migrated from [LUCENE-6322](https://issues.apache.org/jira/browse/LUCENE-6322) by Sekhar, updated May 02 2016
Environment:
```
Windows, JDK 7/8
```

Contributor guide

Open the contributing guide

Research direction

Start with IndexSearcher.doc(int docID, Set fieldsToLoad) and follow CompressingStoredFieldsReader#visitDocument(int docID, StoredFieldVisitor visitor), focusing on how unrequested stored fields are read and decompressed. Compare this with Lucene3xStoredFieldsReader#skipField(); done means large omitted fields are skipped efficiently without unnecessary loading or decompression, while selected fields still load correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.