tikv / tikv/pd

server: make etcd start timeout a no-progress timeout instead of a fixed deadline

Open
#11,014 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

### Background

`server.EtcdStartTimeout` (default `5m`) is currently applied as a **fixed overall deadline** in `(*Server).startEtcd`:

```go
newCtx, cancel := context.WithTimeout(ctx, EtcdStartTimeout)
...
select {
case <-etcd.Server.ReadyNotify():
case <-newCtx.Done():
return errs.ErrCancelStartEtcd.FastGenByArgs()
}
```

The deadline counts down regardless of whether etcd is making progress.

### Problem

When a PD/etcd member restarts and has to catch up a large raft log (applying a backlog of committed entries after a restore/restart), reaching `ReadyNotify()` can legitimately take longer than 5 minutes. In that case:

- `startEtcd` returns `ErrCancelStartEtcd` and the process exits, even though etcd was healthy and steadily making progress;
- under a process supervisor (systemd/k8s), it restarts, replays from the start again, hits the same fixed deadline, and **loops forever** — the node can never come up.

The timeout cannot distinguish a *slow-but-healthy* startup from a *genuine hang* (e.g. a removed member that can never rejoin the quorum).

### Proposal

Treat `EtcdStartTimeout` as a **no-progress (stall) timeout** rather than an absolute deadline:

- Poll `etcd.Server.AppliedIndex()` as a liveness signal while waiting for `ReadyNotify()`.
- Keep waiting as long as the applied index advances (member is applying its raft log → healthy).
- Give up only when there is **no apply progress for `EtcdStartTimeout`** → genuine hang, still fail fast.
- Continue to honor parent `ctx` cancellation for normal shutdown.

`AppliedIndex` is the reliable progress signal for the catch-up phase (committed index may already sit at the target while the state machine is still applying, so it can look "stuck" while healthy).

### Known limitation

This covers the "wait until ready" phase after `embed.StartEtcd` returns. The bulk snapshot load / WAL replay happens **inside** `embed.StartEtcd`, which takes no `ctx` and runs before this wait — so that phase is neither interrupted nor observed by this change. Bounding/observing it would require running `StartEtcd` in a goroutine and is out of scope here.

### Origin / history

The 5-minute etcd start timeout dates back to:

- tikv/pd#1267 — first introduced it as a private const `etcdStartTimeout = time.Minute * 5`, wrapping `startEtcd` in a fixed context deadline to fix a hang where etcd got stuck on startup (port listened but not serving), which also hung the client.
- tikv/pd#1948 — promoted it to the exported var `EtcdStartTimeout = time.Minute * 5` (so tests can override it); the fixed-deadline semantics were carried over unchanged.

Contributor guide

Open the contributing guide

Research direction

Start at (*Server).startEtcd and inspect the existing wait around etcd.Server.ReadyNotify(), including the context created with EtcdStartTimeout. Use AppliedIndex() as the progress signal while preserving parent-context cancellation. Done means slow catch-up continues while the index advances, but startup fails after EtcdStartTimeout without progress; the embed.StartEtcd phase remains out of scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.