ConduitIO / ConduitIO/conduit-connector-postgres

Bug: Logical replication crashes with "walWrite should be >= walFlush" on status update

Open
#312 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Go
Stars
19
Forks
12
Avg merge
12h 20m
Merged PRs (30d)
5

Description

### Bug description

The subscription's `sendStandbyStatusUpdate` crashes with `walWrite (X) should be >= walFlush (Y)`, killing the pipeline. On restart, both values are reset to `StartLSN` and the system recovers — but if Conduit doesn't auto-restart the pipeline, manual server restart is required.

**Environment**

- conduit-connector-postgres v0.13.x / v0.14.x / v0.15.x
- PostgreSQL logical replication with pgoutput proto_version 1

**Impact**

- Pipeline crash on status update interval (every 10s) when the condition is met
- Record replay on restart (up to 10s worth)
- May require manual server restart if Conduit doesn't auto-recover the pipeline

### Root cause

The `sendStandbyStatusUpdate` function treats `walFlushed > walWritten` as a fatal invariant violation:

```go
if walFlushed > s.walWritten {
return fmt.Errorf("walWrite (%s) should be >= walFlush (%s)", s.walWritten, walFlushed)
}
```

Under normal operation, this condition should not occur — `walWritten` is set before the record enters the pipeline, and `Ack` (which sets `walFlushed`) can only be called after the record is processed. The causal chain guarantees `walFlushed <= walWritten`.

However, this condition has been observed in production. The exact trigger is unclear — a likely candidate is a startup edge case where `Ack` is called with a stored position before the subscription produces new records. On restart, both values reset to `StartLSN` and the system recovers, suggesting this is a transient state mismatch rather than a fundamental invariant violation.

The error handling is too aggressive — a recoverable condition is treated as fatal.

### Proposed fix

Downgrade the fatal error to a warning with auto-recovery. This performs the same correction that a restart would (equalizing the two values) without killing the pipeline:

```go
if walFlushed > s.walWritten {
sdk.Logger(ctx).Warn().
Stringer("wal_write", s.walWritten).
Stringer("wal_flush", walFlushed).
Msg("walFlush ahead of walWrite, adjusting walWrite")
s.walWritten = walFlushed
}
```

The `replyWithWALEnd` condition should also use `>=` instead of `==` for defensive handling of this case:

```diff
- replyWithWALEnd := walFlushed == s.walWritten && walFlushed < s.serverWALEnd
+ replyWithWALEnd := walFlushed >= s.walWritten && walFlushed < s.serverWALEnd
```

**Important: do not fix this by advancing `walWritten` for all XLogData messages (Begin, Commit, Relation).** This breaks `replyWithWALEnd` — `walWritten` permanently races ahead of `walFlushed` (which only advances for data records via Ack), making `walFlushed == walWritten` unreachable after any non-data message. The result is unbounded replication slot lag growth on idle or low-traffic slots.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the sendStandbyStatusUpdate function and inspect the walFlushed/walWritten invariant and replyWithWALEnd condition. Reproduce or add coverage for a status update where walFlush is ahead of walWrite, then verify the pipeline logs a warning, adjusts walWrite, and continues without making slot lag grow. Ensure Begin, Commit, and Relation messages do not advance walWritten for this fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.