apache / apache/pinot

Merge/roll-up task takes long time to bootstrap long retention tables

Open
#7,480 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

To bootstrap long retention tables (e.g. 2 year), scheduling 1 time bucket each round will take more than 1 month to process the table. If we can schedule multiple buckets per round, this can be improved.

We may have the issue that spilled over data can't get the chance to be merged and generate more segments than expected. Currently we can fall back to the default behavior if there are segments spanning multiple buckets.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the merge/roll-up task's bootstrap scheduling and its current default fallback behavior. Examine how buckets and spilled-over segments are selected. Done means long-retention tables process multiple buckets per round while preserving merge opportunities, with fallback for segments spanning multiple buckets and tests covering these cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.