Basekick-Labs / Basekick-Labs/arc
feat(metrics): expose replication lag, the signal that actually indicates replication health
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
Arc has no replication lag metric. Lag is the signal every comparable system exposes for replication health, and the one Arc operators currently have no way to see.
Filed alongside #810, where `arc_replication_sequence_gaps_total` was removed rather than implemented: a gap cannot occur silently on a replication connection (the checkpoint sequence check and cumulative payload hash drop the connection first), so the counter measured a condition that fails closed. Lag is the real gap in coverage.
## What every comparable system exposes
| System | Lag signal |
|---|---|
| Kafka | `records-lag-max`; `UnderReplicatedPartitions` as a binary health flag |
| PostgreSQL | `write_lag` / `flush_lag` / `replay_lag` intervals; LSN deltas |
| etcd | `proposals_committed_total − proposals_applied_total` |
| ClickHouse | `log_max_index − log_pointer`; `absolute_delay` seconds |
| InfluxDB Enterprise | hinted-handoff `queueDepth` trend |
None expose a gap counter. All expose lag.
## The data already exists in Arc's protocol
`internal/cluster/replication/protocol.go` already carries both halves:
- `ReplicateSyncAck.CurrentSequence` — "the writer's current sequence number"
- `ReplicateAck.LastSequence` — "the last successfully received and applied sequence"
The receiver tracks its own position in `r.lastSeq` (`receiver.go`). So the delta is available without any protocol change.
## Proposed metrics
```
arc_replication_lag_entries # gauge, per peer: writer CurrentSequence − receiver applied
arc_replication_lag_seconds # gauge, per peer: now − last applied entry's TimestampUS
```
Both are per-peer, so they need a label (`peer` / `node_id`) or one series per connection — worth deciding deliberately, since replication peer counts are small and bounded, unlike measurement or database names.
`ReplicateEntry.TimestampUS` is already on the wire, so the temporal lag needs no protocol change either. Temporal lag is usually the more actionable of the two for operators, because it is comparable across workloads: "the reader is 30 seconds behind" means the same thing at any ingest rate, while "40,000 entries behind" does not.
## A caution worth designing against
CockroachDB shipped a raw positional `behind` metric and then documented that it **cannot be alerted on**: "in a perfectly healthy deployment this number could be in the thousands, especially when some followers are always lagging." They fall back to a binary `ranges_underreplicated` health flag.
So the entries gauge alone is probably not alertable. Worth pairing it with either a seconds-based gauge (which has a workload-independent threshold) or a derived boolean — "this reader has not applied anything in N seconds while the writer advanced" — which is the shape of `UnderReplicatedPartitions`.
`GET /api/v1/cluster` already reports catch-up progress, so part of this may be exposing what the coordinator already knows rather than computing something new. Worth checking that first.
## Test plan
- [ ] Assert lag rises on a reader with replication paused and falls to zero when it catches up
- [ ] Assert the metric is absent, or zero, on a standalone node with no peers rather than reporting a misleading value
- [ ] Confirm the label set cannot grow unbounded (peers are bounded; measurements are not)
- [ ] Regression test must FAIL pre-fix
Contributor guide
Research direction
Start by reading internal/cluster/replication/protocol.go and receiver.go, then check how GET /api/v1/cluster already reports catch-up progress. Decide which lag signals and bounded peer labels to expose, and verify paused-reader lag, catch-up to zero, standalone behavior, bounded labels, and a regression that fails before the fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100