debezium / debezium/dbz

Catch all exception handler for event processing [DBZ-2297]

Open
#339 0 comments 0 reactions 0 assignees View on GitHub
component/mysql-connector migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-2297](https://issues.redhat.com/browse/DBZ-2297)

AFAIK right now there are these configuration options to change how exceptions are handled:
*  [event.processing​.failure.handling.mode|https://debezium.io/documentation/reference/1.2/connectors/mysql.html#mysql-property-event-processing-failure-handling-mode]
* [inconsistent.schema.handling.mode|https://debezium.io/documentation/reference/1.2/connectors/mysql.html#mysql-property-inconsistent-schema-handling-mode]

Also could be considered related 
* [{{database.history.skip.unparseable.ddl}}|https://debezium.io/documentation/reference/1.2/connectors/mysql.html#mysql-property-database-history-skip-unparseable-ddl]

{{Kafka Connect side handlers:}}
{{"errors.tolerance": "all",}}
{{"errors.log.enable":true,}}
{{"errors.log.include.messages":true,}}

{{+ any others?}}

{{Yet it seems it's not enough.}}

For example, when the Debezium internal database schema and MySQL don't match up, then that exception can't be handled gracefully.

{{org.apache.kafka.connect.errors.ConnectException: The binlog event does not contain expected number of columns; the internal schema representation is probably out of sync with the real database schema, or the binlog contains events recorded with binlog_row_image other than FULL or the table in question is an NDB table}}

To reproduce just alter some table with session sql_log_bin=0 and then insert data into that table.

So there are two features that would help operations:
* Catch all configurable exception handler - to tell I'm OK with data loss for only that problematic table, but please keep working on the rest of tables/databases.
* Ability to log out what table/database actually caused that issue.

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Reproduce the failure by altering a table with session sql_log_bin=0 and inserting data, then review the existing event.processing.failure.handling.mode and Kafka Connect error-handler settings. Done means the requested configurable catch-all behavior and table/database identification are defined and verified for the failing case.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, mysql
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.