debezium / debezium/dbz

Incremental snapshot fails fatally when the table schema changes mid-snapshot: recover the chunk instead of stopping the connector

Open
#2,452 0 comments 0 reactions 0 assignees View on GitHub
component/core-library type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

### What Debezium connector do you use and what version?

PostgreSQL connector, current main. Structurally, every relational connector built on `AbstractIncrementalSnapshotChangeEventSource` is affected.

### What behavior do you expect / see?

A DDL statement landing on a table while an incremental snapshot is reading its chunks makes the connector fail fatally. On restart the pending snapshot resumes on the same table, so depending on timing this ranges from one restart per DDL to a permanent crash loop. The mismatch can surface in either direction:

- **column added**: the cached table schema and the result set disagree → `IllegalArgumentException` from `ColumnUtils.toArray` (the message itself hints at this family: *"This might be caused by DBZ-4350"*);
- **column dropped**: with an explicit column projection the chunk query references a column that no longer exists → SQLSTATE `42703` (Postgres) / `42S22` (MySQL family).

This is reachable on current main today: with a column filter configured, `buildProjection` already produces an explicit column list, and the `schemaChanges` suite combined with a `column.exclude.list` fails — as observed during the review of #6000 (DBZ-8430). The same race is what is blocking https://github.com/debezium/debezium/pull/7532 (#2020, generated columns) on the `schemaChanges` suites. Quoting the open question from the #6000 review:

> *"There is a window between the schema change validation step and the execution of the query where I could imagine this error happening, and we should likely guard against this race condition. The question is whether or not we can efficiently guard against this"*

### Proposal

A bounded recovery guard instead of the fatal: on a schema mismatch (either direction), refresh the table schema through the connection, discard the buffered window rows, revert the chunk position, re-read with a fresh projection — with a small bounded attempt counter so a genuinely broken state still fails fast. A DDL landing mid-snapshot then costs one chunk re-read instead of the connector.

We run this guard in production on our fork (built for the parallel-snapshot work in #7362, where the read-only Postgres source's connection-aware schema refresh pattern already pointed the way). A live example — `DROP COLUMN` issued while chunk reads were active on a 484k-row table:

```
12:59:33 incremental snapshot signalled
13:00:05 ALTER TABLE ... DROP COLUMN (snapshot in flight)
13:00:29 WARN Cached schema for table '...' is stale against the database (recovery 1/3):
refreshing schema, reverting chunk and re-reading
13:04:14 snapshot completed (483,989 rows = exact source count)
```

One recovered chunk, no connector failure, exact row count. A subtle detail worth guarding as well: an undefined-column error raised while reading the maximum key can currently be swallowed by the generic catch in `readMaximumKey` and silently skip the table as "empty".

### Relationship with PR #7532

We hope this also helps unblock https://github.com/debezium/debezium/pull/7532: the explicit projection there is correct, and with this guard in place the `schemaChanges` suites should pass with it, since the race window becomes recoverable.

Contributor guide

Open the contributing guide

Research direction

Start at AbstractIncrementalSnapshotChangeEventSource, especially buildProjection and readMaximumKey, then inspect ColumnUtils.toArray and the schemaChanges suite with column.exclude.list. Reproduce the column-add and column-drop races described in the issue. Done means a schema mismatch recovers the chunk with bounded attempts, while a genuinely broken state still fails and readMaximumKey does not silently skip the table.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, postgresql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.