Merge/roll-up task takes long time to bootstrap long retention tables
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
To bootstrap long retention tables (e.g. 2 year), scheduling 1 time bucket each round will take more than 1 month to process the table. If we can schedule multiple buckets per round, this can be improved.
We may have the issue that spilled over data can't get the chance to be merged and generate more segments than expected. Currently we can fall back to the default behavior if there are segments spanning multiple buckets.
Contributor guide
Research direction
Start by tracing the merge/roll-up task's bootstrap scheduling and its current default fallback behavior. Examine how buckets and spilled-over segments are selected. Done means long-retention tables process multiple buckets per round while preserving merge opportunities, with fallback for segments spanning multiple buckets and tests covering these cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100