apache / apache/lucene

Make it possible to throttle (Tiered)MergePolicy when commit rate is high [LUCENE-8162]

Open
#9,210 4 comments 0 reactions 0 assignees View on GitHub
legacy-jira-fix-version:trunk legacy-jira-priority:Major module:core/index type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

As discussed in a recent mailing list thread [1] and observed in a project using Lucene (see [OAK-5192](https://issues.apache.org/jira/browse/OAK-5192) and OAK-6710), it is sometimes helpful to throttle the aggressiveness of (Tiered)MergePolicy when commit rate is high.

In the case of Apache Jackrabbit Oak a dedicated `MergePolicy` was implemented [2].

That MP doesn't merge in case the number of segments is below a certain threshold (e.g. 30) and commit rate (docs per sec and MB per sec) is high (e.g. above 1000 doc / sec , 5MB / sec).

In such impl, the commit rate thresholds adapt to average commit rate by means of single exponential smoothing.

The results in that specific case looked encouraging as it brought a 5% perf improvement in querying and \~10% reduced IO. However Oak has some specifics which might not fit in other scenarios. Anyway it could be interesting to see how this behaves in plain Lucene scenario.

[1] : 

[2] : 

---
Migrated from [LUCENE-8162](https://issues.apache.org/jira/browse/LUCENE-8162) by Tommaso Teofili (@tteofili), 2 votes, updated May 28 2018
Attachments: [LUCENE-8162.0.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8162/LUCENE-8162.0.patch)

Contributor guide

Open the contributing guide

Research direction

Start with Lucene's TieredMergePolicy and the attached LUCENE-8162.0.patch, then review the referenced mailing-list discussion and Oak implementation for context. Determine how throttling should behave in plain Lucene, including segment-count and commit-rate conditions. Done should include an agreed design and validation that the approach improves query performance or reduces I/O without harming other scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.