etcd-io / etcd-io/etcd-operator

Expose CR-configurable per-cluster domain metrics on the operator's Prometheus endpoint

Open
#381 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
164
Forks
72
Avg merge
21h 20m
Merged PRs (30d)
33

Description

TLDR;

The operator already serves a controller-runtime `/metrics` endpoint, but it only exposes generic controller-runtime/Go metrics. Operators running many `EtcdCluster` resources have no first-class, per-cluster signal about the health of the etcd clusters the operator manages (member counts desired vs ready, quorum, leader changes, learner count, TLS state, reconcile timings/errors).

### Proposal

Add an `internal/metrics` package that defines and registers custom Prometheus collectors against the controller-runtime global registry, so they are served on the existing `/metrics` endpoint with no extra HTTP plumbing. Every series is labelled with the cluster `namespace` and `name`.

Proposed metrics (subsystem `etcd_operator_cluster_`):

- `member_count_desired` (gauge) — `spec.size`
- `member_count` (gauge) — members from the member-list API (`status.memberCount`)
- `member_count_ready` (gauge) — StatefulSet `readyReplicas`
- `member_count_healthy` (gauge) — healthy members from status
- `learner_count` (gauge) — non-voting members
- `has_quorum` (gauge 1/0) — healthy members form a strict majority
- `has_leader` (gauge 1/0) — a leader is known
- `tls_enabled` (gauge 1/0) — cluster configured for TLS
- `leader_changes_total` (counter) — observed leader-ID transitions
- `reconcile_duration_seconds` (histogram) — reconcile wall-clock
- `reconcile_errors_total` (counter) — reconciles returning an error

### CR configurability

Add `spec.metrics` to `EtcdCluster`:

- `spec.metrics.enabled` (`*bool`, default true) — toggle/scope the operator domain metrics per cluster; disabling drops that cluster's series.
- `spec.metrics.podMonitor` — when enabled, the operator reconciles a prometheus-operator `PodMonitor` (owned by the `EtcdCluster`) selecting the etcd member pods so Prometheus scrapes etcd's own `/metrics`. Addressed via unstructured so the operator takes no hard dependency on the prometheus-operator module and degrades gracefully (logs + skips) when the CRD is absent.

### Testing

Unit tests register the collectors to an isolated registry, drive state, scrape via `prometheus/client_golang/testutil`, and assert values + labels (including leader-change edge cases and per-cluster series deletion on cluster delete). An e2e test (compile-verified in CI; runs under the live KinD suite) creates a cluster with `spec.metrics.podMonitor` and scrapes the operator `/metrics` endpoint.

I have an implementation ready and will open a PR referencing this issue.

Contributor guide

Open the contributing guide

Research direction

Start with the existing controller-runtime /metrics endpoint and the proposed internal/metrics package, then inspect the EtcdCluster spec and status fields used by reconciliation. Review the planned isolated-registry unit tests and the live KinD e2e test for expected labels, values, deletion behavior, and PodMonitor handling. Done means the listed per-cluster metrics and CR configuration are covered by those tests and served through the existing endpoint.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
infrastructure, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.