confluentinc / confluentinc/ksql
Handle poison pill records by placing them on a dead letter queue
- Dominant language
- Java
- Stars
- 315
- Forks
- 1k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 36
Description
**Is your feature request related to a problem? Please describe.**
We work in an environment where it is possible to encounter poison pill records. This is obviously not intended behaviour but the error condition could happen and it would useful to handle this in a similar fashion to connect.
**Describe the solution you'd like**
For consistency, I think the solution should attempt to mirror as much as possible in how connect handles this (https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/) and expose appropriate JMX metrics to monitor for these errors.
**Describe alternatives you've considered**
There isn't really any alternatives in the context of using ksql
**Additional context**
N/A
Contributor guide
Research direction
No repository files, tests, or entry points are identified in the issue. Start by tracing how ksql currently handles poison pill records and compare the requested behavior with the linked Kafka Connect dead-letter-queue guidance; done means records can be placed on a dead letter queue and the relevant errors expose monitorable JMX metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- observability-sre, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100