SegmentProcessorFramework improvements for merge/rollup
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
Result segments need to be time aligned for some minion tasks (e.g. backfill support for merge/rollup), it will be good if we can add time partition support based on the time window/bucket.
The configs can be something like below:
1. window start time (in milliseconds)
2. time bucket/list of time buckets (in milliseconds)
Say the input n segments have data between 02/2021 to 05/2021, and we want to process all the segments to be monthly aligned, we can set the config as:
```
{
startTime: 02/01/2021 00:00:00,
timeBuckets: [28d, 31d, 30d, 31d]
}
```
Each result segment will only contain data in a certain month.
Contributor guide
Research direction
Start by locating SegmentProcessorFramework and the merge/rollup processing paths. Review how result segments currently select their data, then define completion as support for a window start time and time-bucket configuration that produces time-aligned result segments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100