apache / apache/pinot

[Bug] RTP Tool Should Consider Segment Relocation

Open
#9,211 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, the RTP tool calculates `numCompletedSegmentsPerPartition` as:

`int numCompletedSegmentsPerPartition = (_tableRetentionHours + numHoursToConsume - 1) / numHoursToConsume - 1;`

`_tableRetentionHours` is read from the user-supplied table config. However, this does not work in the scenario where segments are relocated using tag overrides. For example, we move segments from REALTIME to OFFLINE servers every hour. While the table's retention period is 7d, here, `_tableRetentionHours` should be 1h instead of 7d as completed segments reside in REALTIME servers only for about an hour.

@mcvsubbu

Contributor guide

Open the contributing guide

Research direction

Locate the RTP tool code containing the numCompletedSegmentsPerPartition calculation and read how table retention and segment placement are represented. Reproduce the tag-override scenario described in the issue, then verify that the calculation uses the time segments remain on REALTIME servers rather than the table's overall retention period.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.