Incremental snapshot fails fatally when the table schema changes mid-snapshot: recover the chunk instead of stopping the connector
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
### What Debezium connector do you use and what version?
PostgreSQL connector, current main. Structurally, every relational connector built on `AbstractIncrementalSnapshotChangeEventSource` is affected.
### What behavior do you expect / see?
A DDL statement landing on a table while an incremental snapshot is reading its chunks makes the connector fail fatally. On restart the pending snapshot resumes on the same table, so depending on timing this ranges from one restart per DDL to a permanent crash loop. The mismatch can surface in either direction:
- **column added**: the cached table schema and the result set disagree → `IllegalArgumentException` from `ColumnUtils.toArray` (the message itself hints at this family: *"This might be caused by DBZ-4350"*);
- **column dropped**: with an explicit column projection the chunk query references a column that no longer exists → SQLSTATE `42703` (Postgres) / `42S22` (MySQL family).
This is reachable on current main today: with a column filter configured, `buildProjection` already produces an explicit column list, and the `schemaChanges` suite combined with a `column.exclude.list` fails — as observed during the review of #6000 (DBZ-8430). The same race is what is blocking https://github.com/debezium/debezium/pull/7532 (#2020, generated columns) on the `schemaChanges` suites. Quoting the open question from the #6000 review:
> *"There is a window between the schema change validation step and the execution of the query where I could imagine this error happening, and we should likely guard against this race condition. The question is whether or not we can efficiently guard against this"*
### Proposal
A bounded recovery guard instead of the fatal: on a schema mismatch (either direction), refresh the table schema through the connection, discard the buffered window rows, revert the chunk position, re-read with a fresh projection — with a small bounded attempt counter so a genuinely broken state still fails fast. A DDL landing mid-snapshot then costs one chunk re-read instead of the connector.
We run this guard in production on our fork (built for the parallel-snapshot work in #7362, where the read-only Postgres source's connection-aware schema refresh pattern already pointed the way). A live example — `DROP COLUMN` issued while chunk reads were active on a 484k-row table:
```
12:59:33 incremental snapshot signalled
13:00:05 ALTER TABLE ... DROP COLUMN (snapshot in flight)
13:00:29 WARN Cached schema for table '...' is stale against the database (recovery 1/3):
refreshing schema, reverting chunk and re-reading
13:04:14 snapshot completed (483,989 rows = exact source count)
```
One recovered chunk, no connector failure, exact row count. A subtle detail worth guarding as well: an undefined-column error raised while reading the maximum key can currently be swallowed by the generic catch in `readMaximumKey` and silently skip the table as "empty".
### Relationship with PR #7532
We hope this also helps unblock https://github.com/debezium/debezium/pull/7532: the explicit projection there is correct, and with this guard in place the `schemaChanges` suites should pass with it, since the race window becomes recoverable.
Contributor guide
Research direction
Start at AbstractIncrementalSnapshotChangeEventSource, especially buildProjection and readMaximumKey, then inspect ColumnUtils.toArray and the schemaChanges suite with column.exclude.list. Reproduce the column-add and column-drop races described in the issue. Done means a schema mismatch recovers the chunk with bounded attempts, while a genuinely broken state still fails and readMaximumKey does not silently skip the table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, postgresql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100