apache / apache/pinot

Handle poison pills in message Confluent Schema Registry Avro decoder

Open
#9,045 11 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, when bad messages are ingested by Pinot these messages fail decoding and stop consumption of the stream sending the segments to `OFFLINE` state. There is no easy way to recover from this state.

Most of these issues are caused during deserialisation of the binary data into the ingestion format. Some. examples of bad messages included Avro messages with magic byte missing, JSON messages with invalid format, incompatible CSV messages.

The behaviour of the decoder could be governed by config properties of for any reason the current behaviour needs to be kept as is.

Contributor guide

Open the contributing guide

Research direction

Start by locating Pinot's Confluent Schema Registry Avro decoder and the ingestion deserialization path it uses. Determine how poison messages currently stop stream consumption, then define configurable behavior for handling decode failures while preserving the existing behavior when requested; done means bad messages no longer force segments offline under the new setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.