apache / apache/pinot

[Feature] Support time-based backfills

Open
#9,366 1 comment 4 reactions 1 assignee Claimed by @sajjad-moradi View on GitHub
feature in-progress
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

The purpose of this work is to support time-based backfill operations for append-tables. Currently backfills are based on segment name matching: once this feature is added we will be able to backfill periods of time in tables, regardless of whether the segment names match or not.

**Requirements:**

1. Provide support for time-based backfill operations for offline tables and completed segments for real-time tables.
2. This feature should be compatible with Consistent Push and Segment Merge and Rollup.
3. Check that segments uploaded are within the time interval specified.

**Not Supported by this work:**

1. Time-based backfills of data for intervals that intersect segment boundaries are not supported: that is all segments to be replaced must be fully contained in the time interval provided.
2. Backfill of data that span the time interval of consuming segments. What this means is we do not support backfill operations that overlap real time data that has not been turned into completed segments.
3. Merge and rollup operations started during a backfill that could rewind the watermark used for merge and rollups.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.