etcd-io / etcd-io/etcd-operator

Automatic quorum-loss disaster recovery for EtcdCluster

Open
#383 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
164
Forks
72
Avg merge
22h 46m
Merged PRs (30d)
34

Description

## Problem

An etcd cluster makes progress only while a majority (quorum) of its voting members are reachable and can elect a leader. If a majority is **permanently** lost — e.g. two of three members' pods *and* their data volumes are destroyed — the cluster can never elect a leader again on its own, and the operator's normal reconcile loop cannot help: it requeues forever waiting for a leader that will never appear (`couldn't find leader`), and a `member list` against a quorumless cluster fails.

Today the only remedy is a human running etcd's [disaster recovery](https://etcd.io/docs/latest/op-guide/recovery/) procedure by hand.

## Proposal

Teach the operator to **detect** sustained quorum loss and **automatically recover** from a surviving member, following etcd's documented disaster path:

1. Rebuild a single-member cluster from a surviving member's data dir using `--force-new-cluster`.
2. Re-add the remaining members one at a time, reusing the operator's existing learner-add path.

### Safety / guardrails

Recovery is destructive, so detection must be conservative — it should fire **only** on true quorum loss, never on a transient single-member failure:

- A **majority** of expected members must be unreachable with **no leader electable**. A single failed member out of three never triggers recovery (the cluster still has quorum and self-heals normally).
- The condition must **persist for a grace window** before any action — a rolling restart or brief partition that clears within the window is ignored.
- **Size-1 clusters** are never auto-rebuilt (no survivor to recover from; a down single member is an ordinary pod restart).

### Observability

- A new `Recovering` status Condition + a `Recovery` status block exposing the recovery phase (`Detecting` → `Rebuilding` → `ScalingOut` → `Completed`).
- Events at each transition with clear messages.

### Implementation sketch

- Idempotent controller state machine in a self-contained `internal/controller/quorum_recovery.go`, hooked into the health/reconcile path with a single localized branch.
- Unit tests for the detection guard + transitions; an e2e test that breaks a 3-member cluster (delete 2 pods + PVCs) and asserts recovery to a healthy 3-member cluster with data intact.

I have an implementation ready and will open a PR referencing this issue.

Contributor guide

Open the contributing guide

Research direction

Start with the health/reconcile path and the proposed internal/controller/quorum_recovery.go state machine. Review the existing learner-add path and add unit coverage for detection guards and phase transitions, plus the described end-to-end failure scenario. Done means a sustained quorum loss in a three-member cluster recovers to a healthy three-member cluster with data intact, while transient failures and size-1 clusters are not rebuilt.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.