apache / apache/pinot

SegmentProcessorFramework improvements for merge/rollup

Open
#6,979 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Result segments need to be time aligned for some minion tasks (e.g. backfill support for merge/rollup), it will be good if we can add time partition support based on the time window/bucket.

The configs can be something like below:
1. window start time (in milliseconds)
2. time bucket/list of time buckets (in milliseconds)

Say the input n segments have data between 02/2021 to 05/2021, and we want to process all the segments to be monthly aligned, we can set the config as:
```
{
startTime: 02/01/2021 00:00:00,
timeBuckets: [28d, 31d, 30d, 31d]
}
```
Each result segment will only contain data in a certain month.

Contributor guide

Open the contributing guide

Research direction

Start by locating SegmentProcessorFramework and the merge/rollup processing paths. Review how result segments currently select their data, then define completion as support for a window start time and time-bucket configuration that produces time-aligned result segments.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.