pingcap / pingcap/tidb

[PD] Health API can panic with nil pointer dereference during cluster startup/restart

Open
#69,537 4 comments 1 reaction 0 assignees View on GitHub
component/pd contribution may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

Start a small TiDB cluster and poll the PD health API while the cluster is still in a startup/restart window.

The topology used when the panic was observed was:

```text
PD: 1 node
TiKV: 3 nodes
TiDB: 2 nodes
```

A simplified way to trigger the same class of condition is:

```bash
# Terminal 1: start a TiDB cluster with one PD, three TiKV nodes, and two TiDB nodes.
# The exact deployment method should not matter; the important condition is that
# PD has started serving HTTP requests while other cluster/member state may still
# be initializing or reconnecting.

# Terminal 2: poll the PD health API continuously during startup/restart.
while true; do
curl -sS "http://127.0.0.1:2379/pd/api/v1/health" || true
sleep 0.2
done
```

Then restart or bounce cluster components while the health API is still being polled:

```bash
# Restart PD/TiKV/TiDB components, or restart the whole small cluster.
# Keep the health polling loop running while components are rejoining.
```

The panic happened after PD had already reached the point where it was serving requests. The log just before the panic showed the service-discovery/startup phase finishing:

```text
[pd] waiting for DNS: pd
[pd] waiting for DNS: tikv1
[pd] waiting for DNS: tikv2
[pd] waiting for DNS: tikv3
[pd] waiting for DNS: tidb1
[pd] waiting for DNS: tidb2
[pd] DNS ready.
[pd] using binary=/path/to/pd-server
```

The health API request then went through the normal PD HTTP middleware path and entered the PD health handler:

```text
github.com/tikv/pd/server/api.(*healthHandler).GetHealthStatus(...)
```

### 2. What did you expect to see? (Required)

The PD health API should not panic.

If PD is not fully ready, or if member/etcd/cluster state is temporarily unavailable during startup or restart, the health API should return a normal unhealthy response, for example:

```text
HTTP 503 Service Unavailable
```

or another explicit error response.

A health endpoint is normally called by deployment scripts, orchestrators, load balancers, and monitoring systems. It should be safe to call repeatedly, including during startup, restart, and rejoin windows.

### 3. What did you see instead (Required)

PD logged a nil pointer panic while handling the health API request:

```text
[negroni] PANIC: runtime error: invalid memory address or nil pointer dereference

goroutine 517 [running]:
github.com/urfave/negroni/v3.(*Recovery).ServeHTTP.func1()
panic({ ... })

go.etcd.io/etcd/client/v3.(*Client).Ctx(...)
go.etcd.io/etcd/client/v3/client.go:159

github.com/tikv/pd/server/cluster.GetMembers(...)
github.com/tikv/pd/server/cluster/cluster.go:2729

github.com/tikv/pd/server/api.(*healthHandler).GetHealthStatus(...)
github.com/tikv/pd/server/api/health.go:53

github.com/tikv/pd/server/api.(*serviceMiddlewareBuilder).createHandler.WrapFunc.func1(...)
github.com/urfave/negroni/v3.HandlerFunc.ServeHTTP(...)
github.com/tikv/pd/server/api.(*rateLimitMiddleware).ServeHTTP(...)
github.com/tikv/pd/server/api.(*auditMiddleware).ServeHTTP(...)
github.com/tikv/pd/server/api.(*requestInfoMiddleware).ServeHTTP(...)
github.com/gorilla/mux.(*Router).ServeHTTP(...)
```

The relevant panic path is:

```text
healthHandler.GetHealthStatus
-> cluster.GetMembers
-> etcd client Ctx()
-> nil pointer dereference
```

This looks like a missing nil/state check around the PD health handler's member retrieval path. Even if PD is in a transient state, a health API request should not reach a process-level panic.

### 4. What is your TiDB version? (Required)

```text
SELECT tidb_version();

8.0.11-TiDB-v8.5.6
```

The PD stack trace points to the corresponding v8.5.6 PD source path around:

```text
pd/server/api/health.go:53
pd/server/cluster/cluster.go:2729
```

Contributor guide

Open the contributing guide

Research direction

Start at pd/server/api/health.go:53 and follow the healthHandler.GetHealthStatus call into pd/server/cluster/cluster.go:2729 and cluster.GetMembers. Reproduce by polling /pd/api/v1/health during cluster startup or restart. Done means the endpoint never causes a nil-pointer panic and returns an explicit unhealthy or error response when state is unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.