Handle poison pills in message Confluent Schema Registry Avro decoder
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently, when bad messages are ingested by Pinot these messages fail decoding and stop consumption of the stream sending the segments to `OFFLINE` state. There is no easy way to recover from this state.
Most of these issues are caused during deserialisation of the binary data into the ingestion format. Some. examples of bad messages included Avro messages with magic byte missing, JSON messages with invalid format, incompatible CSV messages.
The behaviour of the decoder could be governed by config properties of for any reason the current behaviour needs to be kept as is.
Contributor guide
Research direction
Start by locating Pinot's Confluent Schema Registry Avro decoder and the ingestion deserialization path it uses. Determine how poison messages currently stop stream consumption, then define configurable behavior for handling decode failures while preserving the existing behavior when requested; done means bad messages no longer force segments offline under the new setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100