pingcap / pingcap/tidb

config: cluster table fanout uses invalid :10080 when advertise-address is empty

Open
#68,519 0 comments 0 reactions 1 assignee Claimed by @xhebox View on GitHub
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

1. Deploy a TiDB cluster with TLS enabled (`security.cluster-ssl-ca` set).
2. Trigger a condition where one TiDB node's `advertise-address` is empty at startup (e.g. failback, or `GetLocalIP()` returning `""` before network interfaces are ready).
3. The node registers `ServerInfo{IP: ""}` to etcd.
4. Execute `SELECT * FROM information_schema.CLUSTER_TIDB_TRX` on any TiDB node.
5. Observe slowlog entries with `Query_time: ~40s` and warnings:
```
[:10080] rpc error: ... tls: failed to verify certificate: x509: certificate is valid for ..., not localhost
```

### 2. What did you expect to see? (Required)

- The query should either skip the misconfigured node or fail fast with a clear error.
- TLS authentication handshake failures should not be retried by the backoffer; they indicate a configuration/addressing bug, not a transient network issue.

### 3. What did you see instead? (Required)

The query hangs for exactly ~40s. Root cause chain:

**Step 1: `advertise-address` auto-fill has a gap**

`cmd/tidb-server/main.go:499` fills `AdvertiseAddress` at startup:

```go
if len(cfg.AdvertiseAddress) == 0 && cfg.Host == "0.0.0.0" {
cfg.AdvertiseAddress = util.GetLocalIP() // Can return "" during failback
}
```

If `GetLocalIP()` returns `""`, `AdvertiseAddress` remains empty. The node still registers to etcd:

```go
// pkg/domain/infosync/info.go:1089
info := &ServerInfo{
StaticServerInfo: StaticServerInfo{
IP: cfg.AdvertiseAddress, // ""
},
}
```

**Step 2: Fanout target is built as `:10080` without validation**

Two places construct the gRPC target from `ServerInfo.IP`:

- `infoschema` layer: `net.JoinHostPort(node.IP, StatusPort)` → `:10080` (`tables.go:1888`)
- Coprocessor layer: `net.JoinHostPort(ser.IP, StatusPort)` → `:10080` (`coprocessor.go:597`)

Neither validates that `IP` is non-empty.

**Step 3: gRPC normalizes empty host to `localhost`**

gRPC-Go internally resolves `:10080`:

```go
host, port, _ := net.SplitHostPort(":10080") // host="", port="10080"
if host == "" {
host = "localhost" // authority = "localhost:10080"
}
```

TLS `ClientHandshake` derives `ServerName = "localhost"` from authority. Since TiDB TLS certs do not include `DNS:localhost`, handshake fails immediately.

**Step 4: TLS failure is retried until 40000ms budget is exhausted**

The TLS handshake error is treated as a transient RPC failure and retried by client-go backoffer:

```
tikvRPC backoffer.maxSleep 40000ms is exceeded, errors: send tikv request error: [:10080]...
```

This is a **permanent configuration error**, not a transient network issue.

### 4. What is your TiDB version? (Required)

v8.5.2, reproducible on current master.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.