FixedIntBlockIndexOutput encodes unnecessary integers at the end of a list [LUCENE-2340]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
At closing time, the current FixedIntBlockIndexOutput flushes blocks of blockSize even if there is only a few integers in the block.
This can be problematic and causes a big overhead when using large blockSize (e.g., 1024), on small segments or on rare term posting list.
One solution will be to have a secondary flushBlock method with an additional paramter: the valid length of a buffer. This method will be only called in the FixedIntBlockIndexOutput#close() method.
The way this particular block of integers are encoded are left to subclasses.
---
Migrated from [LUCENE-2340](https://issues.apache.org/jira/browse/LUCENE-2340) by Renaud Delbru, updated May 09 2016
Attachments: [LUCENE-1458-FixedIntBlockIndexOutput.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2340/LUCENE-1458-FixedIntBlockIndexOutput.patch) (versions: 2)
Contributor guide
Research direction
Start by locating FixedIntBlockIndexOutput and its close() path, then inspect flushBlock implementations in its subclasses. Verify how the final partial block is encoded and ensure closing no longer encodes unused integers while preserving the subclasses' encoding behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100