apache / apache/lucene

Date Tiered Compaction

Open
#15,412 8 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

As far as I can tell, Lucene doesn’t currently have a date‑tiered merge policy. I’m considering adding one to better support temporal queries over recent time windows.

On our larger clusters we see frequent paging of index data in and out of the cache. While most users write in temporal order—and TieredMergePolicy often produces a layout similar to what a date‑tiered approach would—it can still create segments with overlapping time ranges. I’m proposing a time‑aware merge policy, similar in spirit to the “date‑tiered compaction” described here: https://engineering.atspotify.com/2014/12/date-tiered-compaction/

I’d appreciate guidance on two points:
1. 1. Is this feasible and something the community would want?
2. Is it worthwhile, or do current features/configurations already achieve something similar?

If this is something interesting to the community, I am happy to write up a design doc and work on this task.

Thanks

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by reviewing Lucene’s existing TieredMergePolicy and related merge-policy configuration, then compare their behavior with the linked date-tiered compaction proposal. Done would require community agreement on feasibility and scope, followed by a design document rather than a narrowly defined code change.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.