cloudnative-pg / cloudnative-pg/klio
WAL client reports unacknowledged flush position to Postgres, breaking RPO 0 guarantee
- Dominant language
- Go
- Stars
- 26
- Forks
- 5
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 113
Description
**Summary**
Klio's WAL streaming client reports a flush LSN back to PostgreSQL that does not reflect data actually durable (fsynced) on the Klio server. This makes it unsafe to configure PostgreSQL synchronous replication (`synchronous_standby_names` +` synchronous_commit`) with Klio as the synchronous target to achieve RPO 0: Postgres can acknowledge commits as durable before the corresponding WAL bytes are guaranteed to survive a Klio server or network failure.
**Technical detail**
- `core/internal/client/sendwal/receiver.go` (`sendFeedback`) sends Postgres a `WALFlushPosition` (and `WALApplyPosition`) derived from `buffer.FlushLSN()`.
- `core/internal/client/sendwal/buffer/buffer.go` sets `flushLSN = writeLSN` immediately after `wal.handler.Write(...)` returns successfully, with no server round-trip involved.
- That `Write` (`buffer/grpc.go`) calls `SendBlock` (`core/internal/client/klioclient/grpcclient/waluploader.go`), which is just `g.innerStream.Send(&PutRequest{...})`, a gRPC client-streaming `Send()`. Per grpc-go's documented contract for `ClientStream.SendMsg`: "SendMsg does not wait until the message is received by the server."
- The only genuine server acknowledgment in this protocol is `CloseAndRecv()` (`grpcclient/connection.go`), which fires once per whole WAL segment (16MB by default), not on the ~200ms cadence (`FlushTimeoutMilliseconds`, `core/pkg/config/client.go`) at which flush feedback is actually sent to Postgres.
- Server-side durability itself is fine (`internal/server/walserver/upload.go` -> `internal/repository/writer.go`'s `DirectWriter.Flush()` does call `file.Sync()`). The gap is purely that the client never learns about it before reporting its own optimistic position upstream.
**Impact**
- With `synchronous_commit = remote_flush` (or `on`) and Klio in `synchronous_standby_names` (its replication connection identifies as `application_name=klio`), Postgres routinely acknowledges commits as durable based on a flush LSN that only reflects "handed to the local send buffer," not "confirmed durable on Klio." A Klio server crash or network partition in that window loses data the application was told was safe, the opposite of RPO 0.
- The same flush position drives the physical replication slot's `restart_lsn` advancement on the Postgres side, so Postgres can be told it is safe to recycle WAL segments Klio was never guaranteed to have durably received. This is a WAL-loss risk independent of synchronous replication.
**Suggested fix direction**
Gate `flushLSN` advancement (and thus both the feedback sent to Postgres and slot `restart_lsn` advancement) on a genuine acknowledgment from the Klio server confirming durable persistence, at a granularity finer than "once per 16MB segment." This likely needs either periodic acks on the Put stream or a bidirectional-streaming protocol change. Evaluate the added per-feedback-interval latency against what the synchronous-replication use case can tolerate.
**Related docs to revisit once resolved**
`documentation/web/docs/user/wal_streaming.md` and the "Synchronous replication" bullet in `documentation/web/docs/user/index.mdx` currently claim zero RPO in synchronous mode.
Contributor guide
Assessment
This issue has not been assessed yet.