config: cluster table fanout uses invalid :10080 when advertise-address is empty
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1. Deploy a TiDB cluster with TLS enabled (`security.cluster-ssl-ca` set).
2. Trigger a condition where one TiDB node's `advertise-address` is empty at startup (e.g. failback, or `GetLocalIP()` returning `""` before network interfaces are ready).
3. The node registers `ServerInfo{IP: ""}` to etcd.
4. Execute `SELECT * FROM information_schema.CLUSTER_TIDB_TRX` on any TiDB node.
5. Observe slowlog entries with `Query_time: ~40s` and warnings:
```
[:10080] rpc error: ... tls: failed to verify certificate: x509: certificate is valid for ..., not localhost
```
### 2. What did you expect to see? (Required)
- The query should either skip the misconfigured node or fail fast with a clear error.
- TLS authentication handshake failures should not be retried by the backoffer; they indicate a configuration/addressing bug, not a transient network issue.
### 3. What did you see instead? (Required)
The query hangs for exactly ~40s. Root cause chain:
**Step 1: `advertise-address` auto-fill has a gap**
`cmd/tidb-server/main.go:499` fills `AdvertiseAddress` at startup:
```go
if len(cfg.AdvertiseAddress) == 0 && cfg.Host == "0.0.0.0" {
cfg.AdvertiseAddress = util.GetLocalIP() // Can return "" during failback
}
```
If `GetLocalIP()` returns `""`, `AdvertiseAddress` remains empty. The node still registers to etcd:
```go
// pkg/domain/infosync/info.go:1089
info := &ServerInfo{
StaticServerInfo: StaticServerInfo{
IP: cfg.AdvertiseAddress, // ""
},
}
```
**Step 2: Fanout target is built as `:10080` without validation**
Two places construct the gRPC target from `ServerInfo.IP`:
- `infoschema` layer: `net.JoinHostPort(node.IP, StatusPort)` → `:10080` (`tables.go:1888`)
- Coprocessor layer: `net.JoinHostPort(ser.IP, StatusPort)` → `:10080` (`coprocessor.go:597`)
Neither validates that `IP` is non-empty.
**Step 3: gRPC normalizes empty host to `localhost`**
gRPC-Go internally resolves `:10080`:
```go
host, port, _ := net.SplitHostPort(":10080") // host="", port="10080"
if host == "" {
host = "localhost" // authority = "localhost:10080"
}
```
TLS `ClientHandshake` derives `ServerName = "localhost"` from authority. Since TiDB TLS certs do not include `DNS:localhost`, handshake fails immediately.
**Step 4: TLS failure is retried until 40000ms budget is exhausted**
The TLS handshake error is treated as a transient RPC failure and retried by client-go backoffer:
```
tikvRPC backoffer.maxSleep 40000ms is exceeded, errors: send tikv request error: [:10080]...
```
This is a **permanent configuration error**, not a transient network issue.
### 4. What is your TiDB version? (Required)
v8.5.2, reproducible on current master.
Contributor guide
Assessment
This issue has not been assessed yet.