debezium / debezium/debezium-design-documents

DDD: Generalize per-table/per-chunk snapshot retry (dbz#2297)

Open
#59 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
38
Forks
31
Avg merge
1d 14h
Merged PRs (30d)
4

Description

## Summary

Proposing a Debezium Design Document to generalize per-table/per-chunk retry during the initial snapshot, addressing debezium/dbz#2297.

Today a single transient `SQLException` on one table or chunk fails the whole initial snapshot. The only recovery, `errors.max.retries`, restarts it from the beginning — re-reading already-completed tables (topic duplicates on large datasets) and unwinding the offset / consistent-read setup.

The Oracle connector already solves this connector-locally via `snapshot.database.errors.max.retries` (narrow ORA-01466 retriability). This DDD proposes lifting the mechanism into the common `RelationalSnapshotChangeEventSource` for all relational connectors, and resolving what a connector-agnostic version must handle that the Oracle-only one could sidestep:

- retriability as an overridable predicate (not "any SQLException"), reconciled with Oracle's existing property;
- connection recovery after an aborted transaction (rollback + reconnect, reapplying connector-specific per-connection state);
- honest duplicate / blast-radius semantics (snapshot rows dispatch per-row mid-scan, so a retry re-emits — reduced blast radius, not idempotent; plus the `SnapshotRecord.FIRST` marker case);
- per-connector consistency (Oracle `AS OF SCN`, MySQL global lock, Postgres exported-snapshot pinning).

Opening this to reserve a DDD number; the full `DDD-.md` PR will follow.

Refs: debezium/dbz#2297

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.