etcd: add a joining PD member as a learner to avoid losing quorum when scaling out from 1 replica
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
PD adds a joining member to the embedded etcd cluster as a **full voter**, before that member's process exists. Growing from exactly 1 replica therefore always loses write quorum until the new member boots and votes.
etcd has supported learners since 3.4 for exactly this case, and the API is already in the `clientv3` PD vendors. PD never calls it. The cost is provably confined to a **single transition**, which makes this a narrow, well-scoped fix.
### Measured: PD v8.5.8, embedded etcd 3.5.15, EKS, TLS enabled
A `1 -> 3` scale-out. Probe is a linearizable `POST /v3/kv/put` against PD's own etcd
client port, 167 ms cadence. **Both conf changes are in one run**, so this is a
controlled A/B of the two edges.
PD's own log (quoted, not derived from the probe):
```
21:31:12.516 [INFO] [cluster.go:421] ["added member"] <- #1, quorum 1->2
21:31:13.054 [WARN] [v3_server.go:920] ["waiting for ReadIndex response took too long, retrying"] x35
21:31:13.932 [WARN] [util.go:170] ["apply request took too long"] ... context deadline exceeded x33
21:31:16.115 [INFO] [server.go:1787] ["no longer a leader because lease has expired"] (+3.60 s)
21:31:17.329 [WARN] [zap_raft.go:85] ["stepped down to follower since quorum is not active"] (+4.81 s)
21:31:25.834 ... "etcdserver: no leader" x201
21:31:30.833 [INFO] [zap_raft.go:77] ["became leader"] (+18.32 s)
21:31:39.724 [INFO] [cluster.go:421] ["added member"] <- #2, quorum 2->3
```
Contributor guide
Research direction
Start at the PD member-add path associated with cluster.go:421 and inspect the vendored clientv3 learner API. Reproduce the 1→3 scale-out with the linearizable POST /v3/kv/put probe, then verify that the joining member is added as a learner before promotion and that quorum is not lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100