apache / apache/lucene

How to control the total number of merging threads when vector data merging easily leads to memory overflow and high CPU cost

Open
#14,554 3 comments 2 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

When there are many shards to merge, vector data merging can easily lead to memory overflow and high CPU cost.
The index.merge.scheduler.max_thread_count parameter can't control the merge thread count, it only pause the writeByte by MergeRateLimiter when the merge thread is bigger then max_thread_count.
But OnHeapHnswGraph has been built during the pause phase, and it will take up so much memory that the Java heap is not enough.
This problem can easily be caused when a datanode with a 32G heap size holds 2-3TB of vector documents(with bbq, the node can contain these data).
The PR https://github.com/apache/lucene/pull/14527 can reduce the heap size, but it don't solve the problem totally.
Is there any solution to this problem?

Contributor guide

Open the contributing guide

Research direction

Start by tracing how index.merge.scheduler.max_thread_count, MergeRateLimiter, and OnHeapHnswGraph interact during vector-data merges. Review the linked Apache Lucene PR 14527 and the surrounding merge-scheduler code to identify where thread control is applied. Done requires an agreed design and validation that limiting concurrent merges prevents the reported heap exhaustion and high CPU cost.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.