How to control the total number of merging threads when vector data merging easily leads to memory overflow and high CPU cost
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
When there are many shards to merge, vector data merging can easily lead to memory overflow and high CPU cost.
The index.merge.scheduler.max_thread_count parameter can't control the merge thread count, it only pause the writeByte by MergeRateLimiter when the merge thread is bigger then max_thread_count.
But OnHeapHnswGraph has been built during the pause phase, and it will take up so much memory that the Java heap is not enough.
This problem can easily be caused when a datanode with a 32G heap size holds 2-3TB of vector documents(with bbq, the node can contain these data).
The PR https://github.com/apache/lucene/pull/14527 can reduce the heap size, but it don't solve the problem totally.
Is there any solution to this problem?
Contributor guide
Research direction
Start by tracing how index.merge.scheduler.max_thread_count, MergeRateLimiter, and OnHeapHnswGraph interact during vector-data merges. Review the linked Apache Lucene PR 14527 and the surrounding merge-scheduler code to identify where thread control is applied. Done requires an agreed design and validation that limiting concurrent merges prevents the reported heap exhaustion and high CPU cost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100