apache / apache/lucene

How to configure TieredMergePolicy for very low segment count?

Open
#14,004 7 comments 1 reaction 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

I have been experimenting with configuring `TieredMergePolicy` to keep the segment count very low:
- segsPerTier = 2
- floorSegmentSize = 512MB

This typically helps if you run queries that have a high per-segment overhead (vector search, multi-term queries) and have a low indexing throughput (especially if indexing and search run on separate hardware so that merges don't disturb searches).

Interestingly, an index that is less than 1GB can still have 10 segments with the above merge policy because of the constraint to not run merges where the resulting segment is less than 50% bigger than the biggest input segment. E.g. consider the following segment sizes: 100kB, 300kB, 800kB, 2MB, 5MB, 12MB, 30MB, 70MB, 150MB, 400MB. There is no pair of segments where the sum is more than 50% bigger than the max input segment.

I have bias against removing this constraint since containing write amplification is important to not run into quadratic merging, but I wonder if there are other ways how we could further reduce the number of segments.

For instance, `TieredMergePolicy` automatically takes the min of `maxMergeAtOnce` and `numSegsPerTier` as a merge factor, but it's not clear to me why this is important. If the merge policy allowed merges to have between 2 and 10 segments in the above example, it could find merges in the described segment structure, and this would likely help have lower write amplification for the same segment count?

Other ideas?

Contributor guide

Open the contributing guide

Research direction

Start by reading the TieredMergePolicy implementation and its handling of segsPerTier, floorSegmentSize, maxMergeAtOnce, and merge selection. Compare the current merge-factor constraint with the example segment sizes and evaluate alternative policies. Done would require an agreed design for reducing segment count without unacceptable write amplification; the issue does not name tests or files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.