apache / apache/pinot

Automatically repair a problematic consuming segment

Open
#15,897 4 comments 0 reactions 0 assignees View on GitHub
ingestion real-time
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

If a Pinot Server encounters multiple consumption errors, it instructs the Controller to mark its replica as OFFLINE in the Ideal State (IS). Currently, the RealtimeSegmentValidationManager (RSVM) periodic job attempts to create a new consuming segment [only if all replicas are in the OFFLINE state](https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L1586-L1587) in the IS.

While this is a useful automation, we have observed many cases where some, but not all, replicas are marked OFFLINE due to transient stream issues. In such cases, all queries are routed to the remaining healthy replicas, which is not ideal.

It would be beneficial if the RSVM job could automatically mitigate these scenarios.

One proposed solution is to issue a force commit when this condition is detected. The force commit should apply only to the affected partition and only if a sufficient number of events have been consumed—e.g., at least half of the desired numRows specified in the segment ZK metadata.

Contributor guide

Open the contributing guide

Research direction

Start with the RealtimeSegmentValidationManager periodic job and the linked condition in PinotLLCRealtimeSegmentManager.java around line 1586. Trace how affected partitions, replica states, force commits, and segment ZK metadata expose the consumed row count. Done means the job handles partially OFFLINE replicas without affecting healthy partitions and enforces the proposed minimum event threshold.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.