Pending incremental snapshot is silently discarded when a boot-time snapshot runs (snapshot.mode=always/configuration_based)
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
### What Debezium connector do you use and what version?
PostgreSQL connector, current main (reproduced on a 3.x-based build). The MySQL/MariaDB family appears structurally affected as well (see scope below).
### What is the connector configuration?
```
snapshot.mode=configuration_based
snapshot.mode.configuration.based.snapshot.data=true
snapshot.mode.configuration.based.snapshot.schema=true
snapshot.mode.configuration.based.start.stream=true
# offsets on a persistent store (any OffsetBackingStore that survives restarts)
```
Also reproducible with `snapshot.mode=always`.
### What behavior do you expect / see?
The incremental snapshots documentation and DDD-3 state that the incremental snapshot context stored in the offsets is enough to resume a pending incremental snapshot after a connector restart. This holds for snapshot modes that skip the boot-time snapshot when offsets exist (`initial`, `no_data`, `when_needed`), but with `always` or `configuration_based` (any variant that actually snapshots) the pending incremental snapshot is **silently discarded** at restart: no resume, no log.
Restart log sequence (connector restarted while an incremental snapshot of `inventory.orders` was in progress):
```
Found previous partition offset ... incremental_snapshot_collections=[{"incremental_snapshot_collections_id":"inventory.orders", ...}]
...boot-time snapshot runs...
Snapshot ended with SnapshotResult [status=COMPLETED, offset=... incrementalSnapshotContext [ ... dataCollectionsToSnapshot=[] ...]]
No incremental snapshot in progress, no action needed on start
```
### Root cause
1. `PostgresOffsetContext.Loader.load()` correctly restores the incremental snapshot context (`AbstractIncrementalSnapshotContext.init()` rebuilds the pending queue, chunk position and maximum key) into `previousOffset`.
2. When the boot-time snapshot runs, `PostgresSnapshotChangeEventSource.determineSnapshotOffset()` builds a fresh offset context via `PostgresOffsetContext.initialContext(...)`, whose constructor always receives a **new, empty** `SignalBased|PostgresReadOnlyIncrementalSnapshotContext`. Only `lastCommitLsn`/`lastCompletelyProcessedLsn` are carried over from `previousOffset` (gated by `Snapshotter.shouldStreamEventsStartingFromSnapshot()`); the incremental context is dropped.
3. `ChangeEventSourceCoordinator` then streams with `snapshotResult.getOffset()`, the fresh context, and the first offset flush persists the loss. The skipped-snapshot path is safe because `SnapshotResult.skipped(previousOffset)` hands the restored offset to streaming unchanged.
4. The `Snapshotter` SPI cannot see the problem: the `snapshotInProgress` flag passed to `shouldSnapshotData/Schema` refers to an unfinished *initial* snapshot only.
### Scope
- PostgreSQL: reproduced and verified (chain above).
- MySQL/MariaDB (binlog family): `MySqlOffsetContext.initial(...)` builds the same fresh incremental context; since the schema is historized there, even schema-only boot snapshots (`no_data`, `recovery`) go through it. Not tested, structurally identical.
### How to reproduce
1. `snapshot.mode=always` (or `configuration_based` with `snapshot.data=true`), offsets on any store that survives restarts.
2. Trigger an incremental snapshot of a table large enough to outlive the test.
3. Kill the connector mid-snapshot (offsets must contain `incremental_snapshot_collections`).
4. Restart: the boot snapshot runs, then streaming starts and the incremental snapshot never resumes, although `Found previous partition offset` shows the restored queue.
### Additional context
Conceptually, `determineSnapshotOffset` already preserves selected state from `previousOffset` across the boot snapshot (the two LSNs); the incremental snapshot context was added to the offsets later and never joined that carry-over. I have a fix ready (carry the restored incremental context into the new offset context: first boot unaffected, no SPI change, no offset format change), field-tested by killing the connector at 30% of an incremental snapshot and observing resume from the exact stored chunk position. PR follows.
Contributor guide
Assessment
This issue has not been assessed yet.