cloudnative-pg / cloudnative-pg/klio
WAL restore: add an end-to-end restore time metric
- Dominant language
- Go
- Stars
- 26
- Forks
- 5
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 113
Description
## Problem
When PostgreSQL asks the plugin for a WAL file (`RESTORE_WAL`, called by
`restore_command`), the plugin already times the whole thing — but only writes it
to the log. There's no metric, so you can't graph it or alert on it.
This hurts most in a replica cluster. The designated primary there keeps up to
date by restoring WAL files from an external Klio source
(`core/internal/cnpgi/wal.go`, the `cluster.IsReplica()` branch; there's an
example in `documentation/web/docs/user/plugin_configuration.md`). So WAL restore
*is* how that cluster replicates, and how fast it runs sets the replica lag. When
such a replica falls behind today, there's no data showing whether the hold-up is
prefetch, the tier, or the network.
The same applies during recovery: restore speed decides how fast a cluster
catches up.
`klio.server.wal.get_duration` doesn't cover this. It only times the server's
side of a single `WAL.Get` call, while the plugin's restore also picks the config
and tier, fails over between tiers, checks the prefetch spool, and renames the
file into place. More important: a prefetch hit never reaches the server — it's
served from the local spool with a rename, so no `WAL.Get` is sent and
`get_duration` sees nothing. Those hits are the normal case while replaying WAL
in order.
## Requirements
- Export the restore time the plugin already measures, as a histogram.
- Tag it with the outcome, which tier served it, and whether it was a prefetch
hit or a real download.
- Keep prefetch hits and misses separable. A hit is a local rename, a miss waits
on the network, so the two are orders of magnitude apart: mixed together, a
percentile just drifts with the hit rate. Kept apart, the hit rate also shows
whether prefetch is keeping up and whether its settings need tuning.
## Verification
- [ ] Every `RESTORE_WAL` is recorded, whether it succeeds or fails.
- [ ] Prefetch hits are distinguishable from downloads, and the serving tier is
visible.
- [ ] The Grafana dashboard shows restore time and rate.
Contributor guide
Research direction
Start at the RESTORE_WAL path in core/internal/cnpgi/wal.go, including the cluster.IsReplica() branch, and trace where the existing restore timing is logged and where prefetch hits and tier selection are handled. Review the Grafana dashboard and the plugin configuration example in documentation/web/docs/user/plugin_configuration.md. Done means every restore records outcome, serving tier, and prefetch hit or download, with the restore time and rate visible in Grafana.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grafana
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100