apache / apache/pinot

Prevent consuming segment from entering ERROR state due to stream connection errors

Open
#11,165 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

## Background
In the event of consumption issues, a consuming segment sends a "stopped consumption" message to the controller, which then marks that consuming segment as OFFLINE in the Ideal State. This approach allows us to recover from transient stream errors through the RealtimeSegmentValidationManager periodic job. This job creates a new consuming segment in the Ideal State if the last segment of a stream partition is in the OFFLINE state.

## Problem Statement
The scenario described in the previous section comes into play after consumption has started, which is when the OFFLINE to CONSUMING transition is completed. During this transition, stream consumers are set up to collect some metadata from the stream. However, if there is any exception thrown while connecting to the stream, the segment goes into the ERROR state in the External View. Unfortunately, this doesn't align with the mentioned automatic recovery process using the RealtimeSegmentValidationManager job.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the OFFLINE to CONSUMING transition and the RealtimeSegmentValidationManager recovery job described in the issue. Inspect how stream connection exceptions affect the segment's External View; done means those errors follow the OFFLINE recovery path instead of leaving the segment in ERROR.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.