[PD] Health API can panic with nil pointer dereference during cluster startup/restart
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
Start a small TiDB cluster and poll the PD health API while the cluster is still in a startup/restart window.
The topology used when the panic was observed was:
```text
PD: 1 node
TiKV: 3 nodes
TiDB: 2 nodes
```
A simplified way to trigger the same class of condition is:
```bash
# Terminal 1: start a TiDB cluster with one PD, three TiKV nodes, and two TiDB nodes.
# The exact deployment method should not matter; the important condition is that
# PD has started serving HTTP requests while other cluster/member state may still
# be initializing or reconnecting.
# Terminal 2: poll the PD health API continuously during startup/restart.
while true; do
curl -sS "http://127.0.0.1:2379/pd/api/v1/health" || true
sleep 0.2
done
```
Then restart or bounce cluster components while the health API is still being polled:
```bash
# Restart PD/TiKV/TiDB components, or restart the whole small cluster.
# Keep the health polling loop running while components are rejoining.
```
The panic happened after PD had already reached the point where it was serving requests. The log just before the panic showed the service-discovery/startup phase finishing:
```text
[pd] waiting for DNS: pd
[pd] waiting for DNS: tikv1
[pd] waiting for DNS: tikv2
[pd] waiting for DNS: tikv3
[pd] waiting for DNS: tidb1
[pd] waiting for DNS: tidb2
[pd] DNS ready.
[pd] using binary=/path/to/pd-server
```
The health API request then went through the normal PD HTTP middleware path and entered the PD health handler:
```text
github.com/tikv/pd/server/api.(*healthHandler).GetHealthStatus(...)
```
### 2. What did you expect to see? (Required)
The PD health API should not panic.
If PD is not fully ready, or if member/etcd/cluster state is temporarily unavailable during startup or restart, the health API should return a normal unhealthy response, for example:
```text
HTTP 503 Service Unavailable
```
or another explicit error response.
A health endpoint is normally called by deployment scripts, orchestrators, load balancers, and monitoring systems. It should be safe to call repeatedly, including during startup, restart, and rejoin windows.
### 3. What did you see instead (Required)
PD logged a nil pointer panic while handling the health API request:
```text
[negroni] PANIC: runtime error: invalid memory address or nil pointer dereference
goroutine 517 [running]:
github.com/urfave/negroni/v3.(*Recovery).ServeHTTP.func1()
panic({ ... })
go.etcd.io/etcd/client/v3.(*Client).Ctx(...)
go.etcd.io/etcd/client/v3/client.go:159
github.com/tikv/pd/server/cluster.GetMembers(...)
github.com/tikv/pd/server/cluster/cluster.go:2729
github.com/tikv/pd/server/api.(*healthHandler).GetHealthStatus(...)
github.com/tikv/pd/server/api/health.go:53
github.com/tikv/pd/server/api.(*serviceMiddlewareBuilder).createHandler.WrapFunc.func1(...)
github.com/urfave/negroni/v3.HandlerFunc.ServeHTTP(...)
github.com/tikv/pd/server/api.(*rateLimitMiddleware).ServeHTTP(...)
github.com/tikv/pd/server/api.(*auditMiddleware).ServeHTTP(...)
github.com/tikv/pd/server/api.(*requestInfoMiddleware).ServeHTTP(...)
github.com/gorilla/mux.(*Router).ServeHTTP(...)
```
The relevant panic path is:
```text
healthHandler.GetHealthStatus
-> cluster.GetMembers
-> etcd client Ctx()
-> nil pointer dereference
```
This looks like a missing nil/state check around the PD health handler's member retrieval path. Even if PD is in a transient state, a health API request should not reach a process-level panic.
### 4. What is your TiDB version? (Required)
```text
SELECT tidb_version();
8.0.11-TiDB-v8.5.6
```
The PD stack trace points to the corresponding v8.5.6 PD source path around:
```text
pd/server/api/health.go:53
pd/server/cluster/cluster.go:2729
```
Contributor guide
Research direction
Start at pd/server/api/health.go:53 and follow the healthHandler.GetHealthStatus call into pd/server/cluster/cluster.go:2729 and cluster.GetMembers. Reproduce by polling /pd/api/v1/health during cluster startup or restart. Done means the endpoint never causes a nil-pointer panic and returns an explicit unhealthy or error response when state is unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100