cloudnative-pg / cloudnative-pg/cloudnative-pg

feat: emit ReplicaCaughtUp event when streaming replica lag drops to zero

Open
#11,292 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
9.3k
Forks
759
Avg merge
2d 6h
Merged PRs (30d)
44

Description

## Problem

When a streaming replica recovers from replication lag — after a network partition, pod restart, or heavy write burst — there is currently no Kubernetes event or signal emitted. Operators must continuously poll `pg_stat_replication` or watch metrics dashboards to know when a replica has fully caught up to the primary. In environments with many clusters and replicas, this creates an observability gap that makes incident response slower and harder to automate.

## Proposed Solution

Emit a `Normal/ReplicaCaughtUp` Kubernetes event on the `Cluster` object the moment a streaming replica's `replay_lag` transitions from non-zero to zero. The event fires exactly once per transition per replica, giving operators a precise, single-fire signal for replication recovery.

This integrates naturally with:
- `kubectl get events` / `kubectl describe cluster`
- Alertmanager and any event-driven alerting pipeline
- Kubernetes event exporters

## Example event output

```
Normal ReplicaCaughtUp cluster/pg-cluster Replica pg-cluster-2 has caught up to the primary
```

## Implementation approach

- Track per-replica lag state in an in-memory map on `ClusterReconciler` (never written to the public CRD status, so the API schema stays clean)
- Call `detectAndEmitReplicaCaughtUpEvents` from `updateClusterStatusThatRequiresInstancesState` on every reconcile loop
- Protect the tracker with a `sync.Mutex` for safe concurrent reconciliation of multiple clusters
- Zero API changes — `InstanceReportedState` and all public types are untouched

A PR with implementation and unit tests is ready: #11291

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.