Implement error handling mechanism for DataException
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
### Feature Request / Improvement
Iceberg Kafka Connector fails when invalid message is received and records are not published to Iceberg or DLQ. To change this behaviour, support for following configurations for handling invalid data exception is implemented:
[errors.tolerance](https://kafka.apache.org/32/generated/sink_connector_config.html#sinkconnectorconfigs_errors.tolerance): Behavior for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behavior to skip over problematic records.
[errors.log.include.messages](https://kafka.apache.org/32/generated/sink_connector_config.html#sinkconnectorconfigs_errors.log.include.messages): Whether to include in the log the Connect record that resulted in a failure. For sink records, the topic, partition, offset, and timestamp will be logged. For source records, the key and value (and their schemas), all headers, and the timestamp, Kafka topic, Kafka partition, source partition, and source offset will be logged. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files.
[errors.deadletterqueue.topic.name](https://kafka.apache.org/32/generated/sink_connector_config.html#sinkconnectorconfigs_errors.deadletterqueue.topic.name): The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, or its transformations or converters. The topic name is blank by default, which means that no messages are to be recorded in the DLQ.
### Query engine
Kafka Connect
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start with the Iceberg Kafka Connector and Kafka Connect error-handling configuration linked in the issue. Check how invalid messages currently affect Iceberg publication and the DLQ, then verify that errors.tolerance, errors.log.include.messages, and errors.deadletterqueue.topic.name produce the requested behavior for invalid records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100