Make it possible to throttle (Tiered)MergePolicy when commit rate is high [LUCENE-8162]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
As discussed in a recent mailing list thread [1] and observed in a project using Lucene (see [OAK-5192](https://issues.apache.org/jira/browse/OAK-5192) and OAK-6710), it is sometimes helpful to throttle the aggressiveness of (Tiered)MergePolicy when commit rate is high.
In the case of Apache Jackrabbit Oak a dedicated `MergePolicy` was implemented [2].
That MP doesn't merge in case the number of segments is below a certain threshold (e.g. 30) and commit rate (docs per sec and MB per sec) is high (e.g. above 1000 doc / sec , 5MB / sec).
In such impl, the commit rate thresholds adapt to average commit rate by means of single exponential smoothing.
The results in that specific case looked encouraging as it brought a 5% perf improvement in querying and \~10% reduced IO. However Oak has some specifics which might not fit in other scenarios. Anyway it could be interesting to see how this behaves in plain Lucene scenario.
[1] :
[2] :
---
Migrated from [LUCENE-8162](https://issues.apache.org/jira/browse/LUCENE-8162) by Tommaso Teofili (@tteofili), 2 votes, updated May 28 2018
Attachments: [LUCENE-8162.0.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8162/LUCENE-8162.0.patch)
Contributor guide
Research direction
Start with Lucene's TieredMergePolicy and the attached LUCENE-8162.0.patch, then review the referenced mailing-list discussion and Oak implementation for context. Determine how throttling should behave in plain Lucene, including segment-count and commit-rate conditions. Done should include an agreed design and validation that the approach improves query performance or reduces I/O without harming other scenarios.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100