cloudnative-pg / cloudnative-pg/cloudnative-pg
feat: emit ReplicaCaughtUp event when streaming replica lag drops to zero
- Dominant language
- Go
- Stars
- 9.3k
- Forks
- 759
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 44
Description
## Problem
When a streaming replica recovers from replication lag — after a network partition, pod restart, or heavy write burst — there is currently no Kubernetes event or signal emitted. Operators must continuously poll `pg_stat_replication` or watch metrics dashboards to know when a replica has fully caught up to the primary. In environments with many clusters and replicas, this creates an observability gap that makes incident response slower and harder to automate.
## Proposed Solution
Emit a `Normal/ReplicaCaughtUp` Kubernetes event on the `Cluster` object the moment a streaming replica's `replay_lag` transitions from non-zero to zero. The event fires exactly once per transition per replica, giving operators a precise, single-fire signal for replication recovery.
This integrates naturally with:
- `kubectl get events` / `kubectl describe cluster`
- Alertmanager and any event-driven alerting pipeline
- Kubernetes event exporters
## Example event output
```
Normal ReplicaCaughtUp cluster/pg-cluster Replica pg-cluster-2 has caught up to the primary
```
## Implementation approach
- Track per-replica lag state in an in-memory map on `ClusterReconciler` (never written to the public CRD status, so the API schema stays clean)
- Call `detectAndEmitReplicaCaughtUpEvents` from `updateClusterStatusThatRequiresInstancesState` on every reconcile loop
- Protect the tracker with a `sync.Mutex` for safe concurrent reconciliation of multiple clusters
- Zero API changes — `InstanceReportedState` and all public types are untouched
A PR with implementation and unit tests is ready: #11291
Contributor guide
Assessment
This issue has not been assessed yet.