server: make etcd start timeout a no-progress timeout instead of a fixed deadline
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
### Background
`server.EtcdStartTimeout` (default `5m`) is currently applied as a **fixed overall deadline** in `(*Server).startEtcd`:
```go
newCtx, cancel := context.WithTimeout(ctx, EtcdStartTimeout)
...
select {
case <-etcd.Server.ReadyNotify():
case <-newCtx.Done():
return errs.ErrCancelStartEtcd.FastGenByArgs()
}
```
The deadline counts down regardless of whether etcd is making progress.
### Problem
When a PD/etcd member restarts and has to catch up a large raft log (applying a backlog of committed entries after a restore/restart), reaching `ReadyNotify()` can legitimately take longer than 5 minutes. In that case:
- `startEtcd` returns `ErrCancelStartEtcd` and the process exits, even though etcd was healthy and steadily making progress;
- under a process supervisor (systemd/k8s), it restarts, replays from the start again, hits the same fixed deadline, and **loops forever** — the node can never come up.
The timeout cannot distinguish a *slow-but-healthy* startup from a *genuine hang* (e.g. a removed member that can never rejoin the quorum).
### Proposal
Treat `EtcdStartTimeout` as a **no-progress (stall) timeout** rather than an absolute deadline:
- Poll `etcd.Server.AppliedIndex()` as a liveness signal while waiting for `ReadyNotify()`.
- Keep waiting as long as the applied index advances (member is applying its raft log → healthy).
- Give up only when there is **no apply progress for `EtcdStartTimeout`** → genuine hang, still fail fast.
- Continue to honor parent `ctx` cancellation for normal shutdown.
`AppliedIndex` is the reliable progress signal for the catch-up phase (committed index may already sit at the target while the state machine is still applying, so it can look "stuck" while healthy).
### Known limitation
This covers the "wait until ready" phase after `embed.StartEtcd` returns. The bulk snapshot load / WAL replay happens **inside** `embed.StartEtcd`, which takes no `ctx` and runs before this wait — so that phase is neither interrupted nor observed by this change. Bounding/observing it would require running `StartEtcd` in a goroutine and is out of scope here.
### Origin / history
The 5-minute etcd start timeout dates back to:
- tikv/pd#1267 — first introduced it as a private const `etcdStartTimeout = time.Minute * 5`, wrapping `startEtcd` in a fixed context deadline to fix a hang where etcd got stuck on startup (port listened but not serving), which also hung the client.
- tikv/pd#1948 — promoted it to the exported var `EtcdStartTimeout = time.Minute * 5` (so tests can override it); the fixed-deadline semantics were carried over unchanged.
Contributor guide
Research direction
Start at (*Server).startEtcd and inspect the existing wait around etcd.Server.ReadyNotify(), including the context created with EtcdStartTimeout. Use AppliedIndex() as the progress signal while preserving parent-context cancellation. Done means slow catch-up continues while the index advances, but startup fails after EtcdStartTimeout without progress; the embed.StartEtcd phase remains out of scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100