tikv / tikv/pd

client: sample repeated member update failure logs

Open
#11,020 1 comment 0 reactions 0 assignees View on GitHub
contribution type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

When TiDB cannot connect to PD, the PD client service-discovery loop can repeatedly
emit an INFO log for every failed membership refresh attempt:

```log
[2026/07/08 22:22:25.978 +08:00] [INFO] [service_discovery.go:896] ["[pd] cannot update member from this url"] [keyspaceName=keyspace1] [url=http://192.168.206.72:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 192.168.206.72:2379: i/o timeout\" target:192.168.206.72:2379 status:TRANSIENT_FAILURE: ..."]
```

During prolonged PD unavailability or network failures, repeated copies of this
per-attempt message can cause a log storm, increase log volume, and obscure more
useful diagnostic information. A complete membership-refresh retry batch remains
necessary for fast recovery. Once that batch confirms both a transport outage
across every current member and a degraded-mode-compatible state for every
corresponding gRPC connection, however, further asynchronous batches add load
without improving recovery.

### Why this becomes a log storm

The message is emitted once for every failed PD URL inside `updateMember()`. A
membership refresh is then multiplied by several retry and scheduling layers:

1. `updateMemberLoop()` executes `updateMember()` through a backoff with a 20 ms
base interval, a 100 ms maximum interval, and a one-second total backoff
budget. When connection failures return quickly, one wake-up can invoke
`updateMember()` about twelve times.
2. Each invocation tries every configured PD URL. With three unreachable PD
endpoints, a single wake-up can therefore emit about 36 copies of the INFO
message before the batch-level warning is logged.
3. Membership checks are not triggered only by the one-minute periodic ticker.
A failed service-mode check schedules one every three seconds. Failed TSO,
region, store, keyspace, meta-storage, resource-manager, router, and HTTP
requests can also call `ScheduleCheckMemberChanged()`.
4. TSO reconnection is a particularly hot path: stream creation and request
failures schedule membership checks while TiDB continues requesting
timestamps for active SQL transactions.
5. `checkMembershipCh` has capacity one and coalesces concurrent notifications,
but it does not rate-limit sustained failures. A new notification can remain
queued while the current retry batch runs, causing another batch to start as
soon as the current one finishes.

Representative TSO call path:

```text
TiDB statement starts or activates a transaction
-> client-go requests a timestamp from the PD client
-> TSO stream creation or request fails
-> ScheduleCheckMemberChanged()
-> updateMemberLoop()
-> Backoffer.Exec(updateMember)
-> updateMember tries every PD URL
-> "[pd] cannot update member from this url"
```

Ordinary PD RPC failures follow a similar path through the common response error
handler. Multiple TiDB instances, keyspaces, and PD client instances multiply
the resulting request and log volume further.

### Design

Preserve the existing complete fast retry batch, then enter a degraded member
refresh mode only when every current member URL fails with a transport error and
every corresponding gRPC connection is observed in `IDLE`, `CONNECTING`, or
`TRANSIENT_FAILURE`.

Here, a transport error means an explicit gRPC dial failure or an RPC failure
classified as `Unavailable` or `DeadlineExceeded` (including a local context
deadline). This classification is only an eligibility signal; the
connection-state condition above is independently required.

While degraded:

- Coalesce asynchronous membership-check requests without issuing
`GetMembers`.
- Inspect local connection state every 100 ms and call `Connect()` for idle
connections.
- Resume the existing retry batch when a connection becomes `READY`, is
missing, is shut down, or the member URL set changes.
- Perform one real membership safety sweep on the existing one-minute tick.
- Keep synchronous `CheckMemberChanged` calls immediate and unsuppressed.

Track transport-failure episodes independently for each service-discovery client
and URL. Log the first detailed error, emit one-minute aggregate summaries for
suppressed errors, and log recovery for the URL that recovers.

### Expected behavior

- Control degraded-mode decisions with structured transport errors and gRPC
connectivity state; do not infer them from error-message text.
- Keep response-header errors, cluster-ID mismatches, application-level errors,
and uncertain failures on the existing retry path.
- Preserve the existing complete retry batch, timeouts, returned errors,
callbacks, and synchronous refresh behavior.
- Avoid additional healthy-mode tickers, background goroutines, configuration,
dependencies, or metrics.
- Reduce repeated asynchronous membership requests and detailed per-URL logs
during sustained transport outages without delaying state-driven recovery.

Contributor guide

Open the contributing guide

Research direction

Start in service_discovery.go at updateMemberLoop() and updateMember(), then trace ScheduleCheckMemberChanged(), CheckMemberChanged(), checkMembershipCh, and Backoffer.Exec(updateMember). Verify transport-error classification and gRPC connectivity states against the stated degraded-mode rules; done means asynchronous failures are coalesced and summarized while synchronous refreshes, recovery, and the existing retry behavior remain intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.