debezium / debezium/debezium-design-documents
DDD-10 Resumable snapshot
- Dominant language
- Python
- Stars
- 38
- Forks
- 31
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 4
Description
The idea is to propose an implementation of resumable initial/blocking snapshot.
Motivation
====
In certain circumstances the requirement for snapshot to finish completely or not at all might be impractical. Such cases might be
- Running cdc in "batch" environment (which is done -- based on my experience and survey results)
- When resuming after an error when snapshotting a very large database
Proposal Pitch
====
Allow snapshots to be partially successful (completed only on subsets of tables). The initial idea is the following
1. Connector is started and originating (the first attempt) snapshot is started and offset is stored with position denoting the start of streaming (O).
2. Snapshot runs and finishes 8/10 tables. Each completed table snapshot is marked in offset (together with source timestamp of the last snapshot record) without updating the log position
3. Connector stops (due to error / timeout / whatever)
4. Connector is started and followup (consecutive) snapshot is started, it discovers a partially completed snapshot, streaming start position is not updated, tables marked as completed are skipped
5. Snapshot runs and finishes the remaining 2 tables.
6. Offset is updated with information that snapshot fully completed, together with the source timestamp of the last produced read event (T)
6. Streaming starts (at this point data might be inconsistent as some updates might lead to past values)
7. Once streaming progresses to an event with source timestamp equal or greater to T a notification singling consistent state is dispatched
Known Limitations
====
Transaction log retention has to be long enough to allow streaming from the originating offset position O. This is not much different from current state (snapshot execution time cannot be greater than log retention), the only difference is that potentially the total snapshot execution time (from 1 to 6) might be longer (depending on the time gap between 3 and 4).
Contributor guide
Assessment
This issue has not been assessed yet.