apache / apache/lucene

Optimize Stored Fields Processing by Avoiding Redundant Loop for Checking Array Element Equality

Open
#12,261 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description
I noticed that when Lucene processes stored fields, it saves `numStoredFields` and `lengths` as shown in the code below:

```java
// save numStoredFields
saveInts(numStoredFields, numBufferedDocs, fieldsStream);

// save lengths
saveInts(lengths, numBufferedDocs, fieldsStream);

private static void saveInts(int[] values, int length, DataOutput out) throws IOException {
if (length == 1) {
out.writeVInt(values[0]);
} else {
StoredFieldsInts.writeInts(values, 0, length, out);
}
}
```
Source: [Lucene90CompressingStoredFieldsWriter.java](https://github.com/apache/lucene/blob/caeabf39309a91997d361b4104bda105d16ae720/lucene/core/src/java/org/apache/lucene/codecs/lucene90/compressing/Lucene90CompressingStoredFieldsWriter.java#L217-L221)

During the `StoredFieldsInts.writeInts(values, 0, length, out);` operation, the method iterates through the `values` array to check if all the elements are the same. I propose an optimization to potentially increase performance by avoiding this loop.

My suggestion is to add two new fields: `numStoredFieldsAllSame` and `docLengthAllSame`. These fields would track whether all the elements in the respective arrays are the same during insertion. By passing these fields to the `writeInts` method, we can eliminate the need for the loop.

Contributor guide

Open the contributing guide

Research direction

Start with Lucene90CompressingStoredFieldsWriter.java, especially saveInts and the insertion paths for numStoredFields and lengths. Then inspect StoredFieldsInts.writeInts to understand its array-equality check and how the proposed flags would affect encoding. Done means the flags are maintained, passed into writeInts, and the redundant equality scan is avoided without changing stored-field output.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.