apache / apache/lucene

Make the perThreadHardLimitMB to be configurable above 2GB

Open
#15,296 5 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

The parameter perThreadHardLimitMB cannot be larger than 2GB, which means a single thread cannot write segments larger than 2GB.
Refer: https://lucene.apache.org/core/9_9_0/core/org/apache/lucene/index/IndexWriterConfig.html#setRAMPerThreadHardLimitMB(int)

This issue proposes to make this parameter configurable above the 2GB limit, so that each thread can write a bigger segment.

When indexing high dimensional vector data, each segment has its own HNSW graph. So more segments mean more graphs to search per shard and more graph rebuild work during merges. With this change, a single indexing thread can flush fewer, and larger segments, which is generally more resource-efficient for vector-heavy workloads.

Contributor guide

Open the contributing guide

Research direction

Begin with Lucene's IndexWriterConfig#setRAMPerThreadHardLimitMB(int) entry point and locate the validation behind the 2GB ceiling. Determine how a larger per-thread limit should be represented and verified; done means values above 2GB are configurable and the setting supports larger segments without breaking existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.