Make the perThreadHardLimitMB to be configurable above 2GB
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
The parameter perThreadHardLimitMB cannot be larger than 2GB, which means a single thread cannot write segments larger than 2GB.
Refer: https://lucene.apache.org/core/9_9_0/core/org/apache/lucene/index/IndexWriterConfig.html#setRAMPerThreadHardLimitMB(int)
This issue proposes to make this parameter configurable above the 2GB limit, so that each thread can write a bigger segment.
When indexing high dimensional vector data, each segment has its own HNSW graph. So more segments mean more graphs to search per shard and more graph rebuild work during merges. With this change, a single indexing thread can flush fewer, and larger segments, which is generally more resource-efficient for vector-heavy workloads.
Contributor guide
Research direction
Begin with Lucene's IndexWriterConfig#setRAMPerThreadHardLimitMB(int) entry point and locate the validation behind the 2GB ceiling. Determine how a larger per-thread limit should be represented and verified; done means values above 2GB are configurable and the setting supports larger segments without breaking existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100