apache / apache/lucene

Should we not enlarge PagedGrowableWriter initial bitPerValue on NodeHash.rehash()?

Open
#12,744 3 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Spawn from https://github.com/apache/lucene/pull/12738

It seems on [rehash](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/fst/NodeHash.java#L310C1-L310C56) we will use an initial bitPerValue set to cover the last node address.

Does that mean every values, including the ones with low-address will use the same bpv as the high-address nodes? PagedGrowableWriter already enlarge the bpv [automatically](https://github.com/apache/lucene/blob/8fa0de2743e87dd264619632978c8dc68323947a/lucene/core/src/java/org/apache/lucene/util/packed/GrowableWriter.java#L86), so maybe we could always set the initial bpv to a small value (like 8?)

Let see if the total size is different when making the change

### Version and environment details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in lucene/core/src/java/org/apache/lucene/util/fst/NodeHash.java at rehash(), then read PagedGrowableWriter and the linked GrowableWriter implementation to understand initial bitPerValue growth. Compare total size with a smaller initial value such as 8 and determine whether the change preserves behavior while reducing memory use.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.