Automatically repair a problematic consuming segment
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
If a Pinot Server encounters multiple consumption errors, it instructs the Controller to mark its replica as OFFLINE in the Ideal State (IS). Currently, the RealtimeSegmentValidationManager (RSVM) periodic job attempts to create a new consuming segment [only if all replicas are in the OFFLINE state](https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L1586-L1587) in the IS.
While this is a useful automation, we have observed many cases where some, but not all, replicas are marked OFFLINE due to transient stream issues. In such cases, all queries are routed to the remaining healthy replicas, which is not ideal.
It would be beneficial if the RSVM job could automatically mitigate these scenarios.
One proposed solution is to issue a force commit when this condition is detected. The force commit should apply only to the affected partition and only if a sufficient number of events have been consumed—e.g., at least half of the desired numRows specified in the segment ZK metadata.
Contributor guide
Research direction
Start with the RealtimeSegmentValidationManager periodic job and the linked condition in PinotLLCRealtimeSegmentManager.java around line 1586. Trace how affected partitions, replica states, force commits, and segment ZK metadata expose the consumed row count. Done means the job handles partially OFFLINE replicas without affecting healthy partitions and enforces the proposed minimum event threshold.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100