etcd-io / etcd-io/etcd-operator
Expose CR-configurable per-cluster domain metrics on the operator's Prometheus endpoint
- Dominant language
- Go
- Stars
- 164
- Forks
- 72
- Avg merge
- 21h 20m
- Merged PRs (30d)
- 33
Description
TLDR;
The operator already serves a controller-runtime `/metrics` endpoint, but it only exposes generic controller-runtime/Go metrics. Operators running many `EtcdCluster` resources have no first-class, per-cluster signal about the health of the etcd clusters the operator manages (member counts desired vs ready, quorum, leader changes, learner count, TLS state, reconcile timings/errors).
### Proposal
Add an `internal/metrics` package that defines and registers custom Prometheus collectors against the controller-runtime global registry, so they are served on the existing `/metrics` endpoint with no extra HTTP plumbing. Every series is labelled with the cluster `namespace` and `name`.
Proposed metrics (subsystem `etcd_operator_cluster_`):
- `member_count_desired` (gauge) — `spec.size`
- `member_count` (gauge) — members from the member-list API (`status.memberCount`)
- `member_count_ready` (gauge) — StatefulSet `readyReplicas`
- `member_count_healthy` (gauge) — healthy members from status
- `learner_count` (gauge) — non-voting members
- `has_quorum` (gauge 1/0) — healthy members form a strict majority
- `has_leader` (gauge 1/0) — a leader is known
- `tls_enabled` (gauge 1/0) — cluster configured for TLS
- `leader_changes_total` (counter) — observed leader-ID transitions
- `reconcile_duration_seconds` (histogram) — reconcile wall-clock
- `reconcile_errors_total` (counter) — reconciles returning an error
### CR configurability
Add `spec.metrics` to `EtcdCluster`:
- `spec.metrics.enabled` (`*bool`, default true) — toggle/scope the operator domain metrics per cluster; disabling drops that cluster's series.
- `spec.metrics.podMonitor` — when enabled, the operator reconciles a prometheus-operator `PodMonitor` (owned by the `EtcdCluster`) selecting the etcd member pods so Prometheus scrapes etcd's own `/metrics`. Addressed via unstructured so the operator takes no hard dependency on the prometheus-operator module and degrades gracefully (logs + skips) when the CRD is absent.
### Testing
Unit tests register the collectors to an isolated registry, drive state, scrape via `prometheus/client_golang/testutil`, and assert values + labels (including leader-change edge cases and per-cluster series deletion on cluster delete). An e2e test (compile-verified in CI; runs under the live KinD suite) creates a cluster with `spec.metrics.podMonitor` and scrapes the operator `/metrics` endpoint.
I have an implementation ready and will open a PR referencing this issue.
Contributor guide
Research direction
Start with the existing controller-runtime /metrics endpoint and the proposed internal/metrics package, then inspect the EtcdCluster spec and status fields used by reconciliation. Review the planned isolated-registry unit tests and the live KinD e2e test for expected labels, values, deletion behavior, and PodMonitor handling. Done means the listed per-cluster metrics and CR configuration are covered by those tests and served through the existing endpoint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, prometheus
- Domain
- infrastructure, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100