etcd-io / etcd-io/etcd-operator
Automatic quorum-loss disaster recovery for EtcdCluster
- Dominant language
- Go
- Stars
- 164
- Forks
- 72
- Avg merge
- 22h 46m
- Merged PRs (30d)
- 34
Description
## Problem
An etcd cluster makes progress only while a majority (quorum) of its voting members are reachable and can elect a leader. If a majority is **permanently** lost — e.g. two of three members' pods *and* their data volumes are destroyed — the cluster can never elect a leader again on its own, and the operator's normal reconcile loop cannot help: it requeues forever waiting for a leader that will never appear (`couldn't find leader`), and a `member list` against a quorumless cluster fails.
Today the only remedy is a human running etcd's [disaster recovery](https://etcd.io/docs/latest/op-guide/recovery/) procedure by hand.
## Proposal
Teach the operator to **detect** sustained quorum loss and **automatically recover** from a surviving member, following etcd's documented disaster path:
1. Rebuild a single-member cluster from a surviving member's data dir using `--force-new-cluster`.
2. Re-add the remaining members one at a time, reusing the operator's existing learner-add path.
### Safety / guardrails
Recovery is destructive, so detection must be conservative — it should fire **only** on true quorum loss, never on a transient single-member failure:
- A **majority** of expected members must be unreachable with **no leader electable**. A single failed member out of three never triggers recovery (the cluster still has quorum and self-heals normally).
- The condition must **persist for a grace window** before any action — a rolling restart or brief partition that clears within the window is ignored.
- **Size-1 clusters** are never auto-rebuilt (no survivor to recover from; a down single member is an ordinary pod restart).
### Observability
- A new `Recovering` status Condition + a `Recovery` status block exposing the recovery phase (`Detecting` → `Rebuilding` → `ScalingOut` → `Completed`).
- Events at each transition with clear messages.
### Implementation sketch
- Idempotent controller state machine in a self-contained `internal/controller/quorum_recovery.go`, hooked into the health/reconcile path with a single localized branch.
- Unit tests for the detection guard + transitions; an e2e test that breaks a 3-member cluster (delete 2 pods + PVCs) and asserts recovery to a healthy 3-member cluster with data intact.
I have an implementation ready and will open a PR referencing this issue.
Contributor guide
Research direction
Start with the health/reconcile path and the proposed internal/controller/quorum_recovery.go state machine. Review the existing learner-add path and add unit coverage for detection guards and phase transitions, plus the described end-to-end failure scenario. Done means a sustained quorum loss in a three-member cluster recovers to a healthy three-member cluster with data intact, while transient failures and size-1 clusters are not rebuilt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 30/100