tikv / tikv/pd

etcd: add a joining PD member as a learner to avoid losing quorum when scaling out from 1 replica

Open
#11,236 0 comments 0 reactions 0 assignees View on GitHub
contribution type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task
PD adds a joining member to the embedded etcd cluster as a **full voter**, before that member's process exists. Growing from exactly 1 replica therefore always loses write quorum until the new member boots and votes.
etcd has supported learners since 3.4 for exactly this case, and the API is already in the `clientv3` PD vendors. PD never calls it. The cost is provably confined to a **single transition**, which makes this a narrow, well-scoped fix.

### Measured: PD v8.5.8, embedded etcd 3.5.15, EKS, TLS enabled

A `1 -> 3` scale-out. Probe is a linearizable `POST /v3/kv/put` against PD's own etcd
client port, 167 ms cadence. **Both conf changes are in one run**, so this is a
controlled A/B of the two edges.

PD's own log (quoted, not derived from the probe):

```
21:31:12.516 [INFO] [cluster.go:421] ["added member"] <- #1, quorum 1->2
21:31:13.054 [WARN] [v3_server.go:920] ["waiting for ReadIndex response took too long, retrying"] x35
21:31:13.932 [WARN] [util.go:170] ["apply request took too long"] ... context deadline exceeded x33
21:31:16.115 [INFO] [server.go:1787] ["no longer a leader because lease has expired"] (+3.60 s)
21:31:17.329 [WARN] [zap_raft.go:85] ["stepped down to follower since quorum is not active"] (+4.81 s)
21:31:25.834 ... "etcdserver: no leader" x201
21:31:30.833 [INFO] [zap_raft.go:77] ["became leader"] (+18.32 s)
21:31:39.724 [INFO] [cluster.go:421] ["added member"] <- #2, quorum 2->3
```

Contributor guide

Open the contributing guide

Research direction

Start at the PD member-add path associated with cluster.go:421 and inspect the vendored clientv3 learner API. Reproduce the 1→3 scale-out with the linearizable POST /v3/kv/put probe, then verify that the joining member is added as a learner before promotion and that quorum is not lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.