apache / apache/lucene

FixedIntBlockIndexOutput encodes unnecessary integers at the end of a list [LUCENE-2340]

Open
#3,416 6 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0-ALPHA legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-label:dead legacy-jira-priority:Minor module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

At closing time, the current FixedIntBlockIndexOutput flushes blocks of blockSize even if there is only a few integers in the block.
This can be problematic and causes a big overhead when using large blockSize (e.g., 1024), on small segments or on rare term posting list.

One solution will be to have a secondary flushBlock method with an additional paramter: the valid length of a buffer. This method will be only called in the FixedIntBlockIndexOutput#close() method.
The way this particular block of integers are encoded are left to subclasses.

---
Migrated from [LUCENE-2340](https://issues.apache.org/jira/browse/LUCENE-2340) by Renaud Delbru, updated May 09 2016
Attachments: [LUCENE-1458-FixedIntBlockIndexOutput.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2340/LUCENE-1458-FixedIntBlockIndexOutput.patch) (versions: 2)

Contributor guide

Open the contributing guide

Research direction

Start by locating FixedIntBlockIndexOutput and its close() path, then inspect flushBlock implementations in its subclasses. Verify how the final partial block is encoded and ensure closing no longer encodes unused integers while preserving the subclasses' encoding behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.