debezium / debezium/debezium-design-documents
DDD: Generalize per-table/per-chunk snapshot retry (dbz#2297)
- Dominant language
- Python
- Stars
- 38
- Forks
- 31
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 4
Description
## Summary
Proposing a Debezium Design Document to generalize per-table/per-chunk retry during the initial snapshot, addressing debezium/dbz#2297.
Today a single transient `SQLException` on one table or chunk fails the whole initial snapshot. The only recovery, `errors.max.retries`, restarts it from the beginning — re-reading already-completed tables (topic duplicates on large datasets) and unwinding the offset / consistent-read setup.
The Oracle connector already solves this connector-locally via `snapshot.database.errors.max.retries` (narrow ORA-01466 retriability). This DDD proposes lifting the mechanism into the common `RelationalSnapshotChangeEventSource` for all relational connectors, and resolving what a connector-agnostic version must handle that the Oracle-only one could sidestep:
- retriability as an overridable predicate (not "any SQLException"), reconciled with Oracle's existing property;
- connection recovery after an aborted transaction (rollback + reconnect, reapplying connector-specific per-connection state);
- honest duplicate / blast-radius semantics (snapshot rows dispatch per-row mid-scan, so a retry re-emits — reduced blast radius, not idempotent; plus the `SnapshotRecord.FIRST` marker case);
- per-connector consistency (Oracle `AS OF SCN`, MySQL global lock, Postgres exported-snapshot pinning).
Opening this to reserve a DDD number; the full `DDD-.md` PR will follow.
Refs: debezium/dbz#2297
Contributor guide
Assessment
This issue has not been assessed yet.