Catch all exception handler for event processing [DBZ-2297]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-2297](https://issues.redhat.com/browse/DBZ-2297)
AFAIK right now there are these configuration options to change how exceptions are handled:
* [event.processing.failure.handling.mode|https://debezium.io/documentation/reference/1.2/connectors/mysql.html#mysql-property-event-processing-failure-handling-mode]
* [inconsistent.schema.handling.mode|https://debezium.io/documentation/reference/1.2/connectors/mysql.html#mysql-property-inconsistent-schema-handling-mode]
Also could be considered related
* [{{database.history.skip.unparseable.ddl}}|https://debezium.io/documentation/reference/1.2/connectors/mysql.html#mysql-property-database-history-skip-unparseable-ddl]
{{Kafka Connect side handlers:}}
{{"errors.tolerance": "all",}}
{{"errors.log.enable":true,}}
{{"errors.log.include.messages":true,}}
{{+ any others?}}
{{Yet it seems it's not enough.}}
For example, when the Debezium internal database schema and MySQL don't match up, then that exception can't be handled gracefully.
{{org.apache.kafka.connect.errors.ConnectException: The binlog event does not contain expected number of columns; the internal schema representation is probably out of sync with the real database schema, or the binlog contains events recorded with binlog_row_image other than FULL or the table in question is an NDB table}}
To reproduce just alter some table with session sql_log_bin=0 and then insert data into that table.
So there are two features that would help operations:
* Catch all configurable exception handler - to tell I'm OK with data loss for only that problematic table, but please keep working on the rest of tables/databases.
* Ability to log out what table/database actually caused that issue.
Contributor guide
Research direction
No source files, tests, or entry points are named. Reproduce the failure by altering a table with session sql_log_bin=0 and inserting data, then review the existing event.processing.failure.handling.mode and Kafka Connect error-handler settings. Done means the requested configurable catch-all behavior and table/database identification are defined and verified for the failing case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, mysql
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100