Source offsets acknowledged to the database can run ahead of the persisted offsets, breaking recovery after a crash
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
After a hard crash (SIGKILL, OOM), a Postgres connector can permanently fail to restart with:
The connector is trying to read change stream starting at ... but this is no longer available on the
server. Reconfigure the connector to use a snapshot mode when needed.
even though the replication slot still exists and was never recreated: the stored offset LSN sits slightly
below the slot's `restart_lsn`, so the WAL it points to has been recycled.
We hit this on three connectors sharing one PostgreSQL instance, which all crashed hard within the same
weekend: in each case the persisted offset was between a few hundred KB and ~55MB behind the slot's
`restart_lsn`, and its timestamp matched the last successful offset flush before the crash.
Root cause: the runtime side is correctly ordered (both `AsyncEmbeddedEngine` and the Kafka Connect worker
call `SourceTask#commit()` only after a successful offset flush), but `BaseSourceTask#performCommit()` then
acknowledges `lastOffsets`, the most recent in-memory offsets recorded by `poll()`. By the time the
notification arrives, those can be up to a full offset flush interval ahead of what was durably persisted.
The acknowledgment flows into `StreamingChangeEventSource#commitOffset`, whose contract is to "indicate how
far the source TX log can be discarded": the connector therefore authorizes discarding WAL it may still
need. On Postgres this maps to `setFlushedLSN` and the standby status update, whose field is defined by the
protocol as the position "flushed to disk" on the client, which an in-memory position is not.
In steady state the mismatch is invisible: on a clean restart the WAL below the acknowledged position is
still retained. The failure needs a hard crash inside the window where the slot's `restart_lsn` has already
moved past the persisted offset; with the default 60s flush interval that window reopens on every cycle.
The same applies to any offset store (file, Kafka, custom) and to every connector implementing
`commitOffset` (Postgres, Oracle, SQL Server); connectors with the default no-op are unaffected.
The direction of a fix follows from the asymmetry of the two errors: under-acknowledging costs at most one
extra flush interval of source-side retention and heals on the next cycle, while over-acknowledging can
lose data irrecoverably. Proposal: acknowledge, at each `commit()` notification, the offsets captured at
the previous notification. Everything captured then is provably covered by the flush that triggered the
current notification, so the acknowledged position can never run ahead of the store. The cost is a source
acknowledgment lagging one flush interval behind.
Validated with an invariant probe on a live incremental snapshot over a 17.5M-row table: with the fix, the
slot's `confirmed_flush_lsn` never exceeded the persisted offset across the whole run (before the fix the
violation reappeared on every flush cycle), and a SIGKILL mid-snapshot was followed by a clean restart and
an exact resume.
Contributor guide
Assessment
This issue has not been assessed yet.