confluentinc / confluentinc/ksql

Handle poison pill records by placing them on a dead letter queue

Open
#6,563 0 comments 17 reactions 0 assignees View on GitHub
enhancement epic
Dominant language
Java
Stars
315
Forks
1k
Avg merge
1d 7h
Merged PRs (30d)
36

Description

**Is your feature request related to a problem? Please describe.**
We work in an environment where it is possible to encounter poison pill records. This is obviously not intended behaviour but the error condition could happen and it would useful to handle this in a similar fashion to connect.

**Describe the solution you'd like**
For consistency, I think the solution should attempt to mirror as much as possible in how connect handles this (https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/) and expose appropriate JMX metrics to monitor for these errors.

**Describe alternatives you've considered**
There isn't really any alternatives in the context of using ksql

**Additional context**
N/A

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or entry points are identified in the issue. Start by tracing how ksql currently handles poison pill records and compare the requested behavior with the linked Kafka Connect dead-letter-queue guidance; done means records can be placed on a dead letter queue and the relevant errors expose monitorable JMX metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
observability-sre, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.