apache / apache/lucene

Estimate memory usage for merges

Open
#14,225 1 comment 1 reaction 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Merges should expose an estimation of the amount of heap memory that will be used for the actual merge to be done.

This information should be exposed as part of the merge information, so merge schedulers can check the amount of memory needed to perform the merge and thus make decisions on throttling merges that would take too much memory.

As #14208 describes, this is specially important for structures like HNSW graphs that can take a huge amount of memory during merges.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by locating the merge information exposed to merge schedulers and read the context from #14208, especially the HNSW merge case. Done means merge information includes a heap-memory estimate that schedulers can use when making throttling decisions, with tests covering the estimate.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.