debezium / debezium/dbz

Pending incremental snapshot is silently discarded when a boot-time snapshot runs (snapshot.mode=always/configuration_based)

Open
#2,325 13 comments 0 reactions 0 assignees View on GitHub
component/mariadb-connector component/mysql-connector component/postgresql-connector type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

### What Debezium connector do you use and what version?

PostgreSQL connector, current main (reproduced on a 3.x-based build). The MySQL/MariaDB family appears structurally affected as well (see scope below).

### What is the connector configuration?

```
snapshot.mode=configuration_based
snapshot.mode.configuration.based.snapshot.data=true
snapshot.mode.configuration.based.snapshot.schema=true
snapshot.mode.configuration.based.start.stream=true
# offsets on a persistent store (any OffsetBackingStore that survives restarts)
```

Also reproducible with `snapshot.mode=always`.

### What behavior do you expect / see?

The incremental snapshots documentation and DDD-3 state that the incremental snapshot context stored in the offsets is enough to resume a pending incremental snapshot after a connector restart. This holds for snapshot modes that skip the boot-time snapshot when offsets exist (`initial`, `no_data`, `when_needed`), but with `always` or `configuration_based` (any variant that actually snapshots) the pending incremental snapshot is **silently discarded** at restart: no resume, no log.

Restart log sequence (connector restarted while an incremental snapshot of `inventory.orders` was in progress):

```
Found previous partition offset ... incremental_snapshot_collections=[{"incremental_snapshot_collections_id":"inventory.orders", ...}]
...boot-time snapshot runs...
Snapshot ended with SnapshotResult [status=COMPLETED, offset=... incrementalSnapshotContext [ ... dataCollectionsToSnapshot=[] ...]]
No incremental snapshot in progress, no action needed on start
```

### Root cause

1. `PostgresOffsetContext.Loader.load()` correctly restores the incremental snapshot context (`AbstractIncrementalSnapshotContext.init()` rebuilds the pending queue, chunk position and maximum key) into `previousOffset`.
2. When the boot-time snapshot runs, `PostgresSnapshotChangeEventSource.determineSnapshotOffset()` builds a fresh offset context via `PostgresOffsetContext.initialContext(...)`, whose constructor always receives a **new, empty** `SignalBased|PostgresReadOnlyIncrementalSnapshotContext`. Only `lastCommitLsn`/`lastCompletelyProcessedLsn` are carried over from `previousOffset` (gated by `Snapshotter.shouldStreamEventsStartingFromSnapshot()`); the incremental context is dropped.
3. `ChangeEventSourceCoordinator` then streams with `snapshotResult.getOffset()`, the fresh context, and the first offset flush persists the loss. The skipped-snapshot path is safe because `SnapshotResult.skipped(previousOffset)` hands the restored offset to streaming unchanged.
4. The `Snapshotter` SPI cannot see the problem: the `snapshotInProgress` flag passed to `shouldSnapshotData/Schema` refers to an unfinished *initial* snapshot only.

### Scope

- PostgreSQL: reproduced and verified (chain above).
- MySQL/MariaDB (binlog family): `MySqlOffsetContext.initial(...)` builds the same fresh incremental context; since the schema is historized there, even schema-only boot snapshots (`no_data`, `recovery`) go through it. Not tested, structurally identical.

### How to reproduce

1. `snapshot.mode=always` (or `configuration_based` with `snapshot.data=true`), offsets on any store that survives restarts.
2. Trigger an incremental snapshot of a table large enough to outlive the test.
3. Kill the connector mid-snapshot (offsets must contain `incremental_snapshot_collections`).
4. Restart: the boot snapshot runs, then streaming starts and the incremental snapshot never resumes, although `Found previous partition offset` shows the restored queue.

### Additional context

Conceptually, `determineSnapshotOffset` already preserves selected state from `previousOffset` across the boot snapshot (the two LSNs); the incremental snapshot context was added to the offsets later and never joined that carry-over. I have a fix ready (carry the restored incremental context into the new offset context: first boot unaffected, no SPI change, no offset format change), field-tested by killing the connector at 30% of an incremental snapshot and observing resume from the exact stored chunk position. PR follows.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.