ConduitIO / ConduitIO/conduit-connector-postgres
Bug: Logical replication crashes with "walWrite should be >= walFlush" on status update
- Dominant language
- Go
- Stars
- 19
- Forks
- 12
- Avg merge
- 12h 20m
- Merged PRs (30d)
- 5
Description
### Bug description
The subscription's `sendStandbyStatusUpdate` crashes with `walWrite (X) should be >= walFlush (Y)`, killing the pipeline. On restart, both values are reset to `StartLSN` and the system recovers — but if Conduit doesn't auto-restart the pipeline, manual server restart is required.
**Environment**
- conduit-connector-postgres v0.13.x / v0.14.x / v0.15.x
- PostgreSQL logical replication with pgoutput proto_version 1
**Impact**
- Pipeline crash on status update interval (every 10s) when the condition is met
- Record replay on restart (up to 10s worth)
- May require manual server restart if Conduit doesn't auto-recover the pipeline
### Root cause
The `sendStandbyStatusUpdate` function treats `walFlushed > walWritten` as a fatal invariant violation:
```go
if walFlushed > s.walWritten {
return fmt.Errorf("walWrite (%s) should be >= walFlush (%s)", s.walWritten, walFlushed)
}
```
Under normal operation, this condition should not occur — `walWritten` is set before the record enters the pipeline, and `Ack` (which sets `walFlushed`) can only be called after the record is processed. The causal chain guarantees `walFlushed <= walWritten`.
However, this condition has been observed in production. The exact trigger is unclear — a likely candidate is a startup edge case where `Ack` is called with a stored position before the subscription produces new records. On restart, both values reset to `StartLSN` and the system recovers, suggesting this is a transient state mismatch rather than a fundamental invariant violation.
The error handling is too aggressive — a recoverable condition is treated as fatal.
### Proposed fix
Downgrade the fatal error to a warning with auto-recovery. This performs the same correction that a restart would (equalizing the two values) without killing the pipeline:
```go
if walFlushed > s.walWritten {
sdk.Logger(ctx).Warn().
Stringer("wal_write", s.walWritten).
Stringer("wal_flush", walFlushed).
Msg("walFlush ahead of walWrite, adjusting walWrite")
s.walWritten = walFlushed
}
```
The `replyWithWALEnd` condition should also use `>=` instead of `==` for defensive handling of this case:
```diff
- replyWithWALEnd := walFlushed == s.walWritten && walFlushed < s.serverWALEnd
+ replyWithWALEnd := walFlushed >= s.walWritten && walFlushed < s.serverWALEnd
```
**Important: do not fix this by advancing `walWritten` for all XLogData messages (Begin, Commit, Relation).** This breaks `replyWithWALEnd` — `walWritten` permanently races ahead of `walFlushed` (which only advances for data records via Ack), making `walFlushed == walWritten` unreachable after any non-data message. The result is unbounded replication slot lag growth on idle or low-traffic slots.
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the sendStandbyStatusUpdate function and inspect the walFlushed/walWritten invariant and replyWithWALEnd condition. Reproduce or add coverage for a status update where walFlush is ahead of walWrite, then verify the pipeline logs a warning, adjusts walWrite, and continues without making slot lag grow. Ensure Begin, Commit, and Relation messages do not advance walWritten for this fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100