apache / apache/lucene

Optimize StoredFields merge strategy based on chunk deletion ratio

Open
#16,045 0 comments 1 reaction 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

For StoredField merging, the merge strategy has to switch from the `BULK` mode to the `DOC` mode even if only a document is deleted, which significantly increases merging overhead.
https://github.com/apache/lucene/blob/2dee50e21f868aaae436c1dc47d6678564d69b4d/lucene/core/src/java/org/apache/lucene/codecs/lucene90/compressing/Lucene90CompressingStoredFieldsWriter.java#L684

StoredField is compressed at the chunk granularity I propose introducing a new merge strategy:

- For the chunk with no deleted documents, retain `copyChunk `during merging;

- For the chunk containing deleted documents, fall back to `copyOneDoc`.

Furthermore, we can leverage the segment deletion ratio (e.g., a threshold below 5%) as a condition to enable or disable this optimized merging logic.

Contributor guide

Open the contributing guide

Research direction

Start in lucene/core/src/java/org/apache/lucene/codecs/lucene90/compressing/Lucene90CompressingStoredFieldsWriter.java around line 684 and trace the current BULK/DOC merge selection. Confirm the intended behavior for chunks with and without deletions, including the deletion-ratio threshold, and use the existing stored-fields merge tests or benchmarks if found nearby to verify the optimization.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.