debezium / debezium/dbz

Incomplete incremental snapshots and retained transactions after schema mismatches in MySQL and MariaDB

Open
#2,604 0 comments 0 reactions 0 assignees View on GitHub
component/mariadb-connector component/mysql-connector type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Problem

During incremental snapshots, concurrent schema changes can leave query results out of sync with the connector's cached table definition. A missing-column mismatch can cause the current table to be skipped and bypass the normal window-close path.

In MySQL and MariaDB, this can leave snapshot rows unprocessed, prevent progress to subsequent tables, and retain an open read transaction whose metadata locks block DDL.

The failure affects both read-only and signal-based incremental snapshots. Releasing the transaction alone does not restore the skipped snapshot work.

## Reproduction

Observed with Debezium 3.7.0-SNAPSHOT, MySQL 8.4.11, and MariaDB 11.4.13, with `incremental.snapshot.allow.schema.changes=true`.

A deterministic component reproduction uses real JDBC connections while controlling schema-cache timing and watermark callbacks:

1. Prepare an incremental snapshot of two tables.
2. For the first table, use a cached definition that lacks a column present in the query result, while the snapshot schema-verification state matches that result.
3. Read a chunk to trigger the missing-column mismatch.
4. Check whether the snapshot continues through both tables and whether an `ALTER TABLE` from another connection is blocked by the retained transaction.

## Expected behavior

Retry the affected chunk in a subsequent watermark window instead of abandoning the table. Preserve snapshot progress and allow streaming to process pending schema changes before retrying.

Release chunk transactions on failure and surface recovery failures through the connector's existing error handling, without silently skipping snapshot work or reporting false completion.

Schema-mismatch recovery is scoped to MySQL and MariaDB read-only and signal-based incremental snapshots. Signal-table configuration validation and recovery for other connectors are separate concerns.

Related PR: [debezium/debezium#8004](https://github.com/debezium/debezium/pull/8004)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.