client: sample repeated member update failure logs
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
When TiDB cannot connect to PD, the PD client service-discovery loop can repeatedly
emit an INFO log for every failed membership refresh attempt:
```log
[2026/07/08 22:22:25.978 +08:00] [INFO] [service_discovery.go:896] ["[pd] cannot update member from this url"] [keyspaceName=keyspace1] [url=http://192.168.206.72:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 192.168.206.72:2379: i/o timeout\" target:192.168.206.72:2379 status:TRANSIENT_FAILURE: ..."]
```
During prolonged PD unavailability or network failures, repeated copies of this
per-attempt message can cause a log storm, increase log volume, and obscure more
useful diagnostic information. A complete membership-refresh retry batch remains
necessary for fast recovery. Once that batch confirms both a transport outage
across every current member and a degraded-mode-compatible state for every
corresponding gRPC connection, however, further asynchronous batches add load
without improving recovery.
### Why this becomes a log storm
The message is emitted once for every failed PD URL inside `updateMember()`. A
membership refresh is then multiplied by several retry and scheduling layers:
1. `updateMemberLoop()` executes `updateMember()` through a backoff with a 20 ms
base interval, a 100 ms maximum interval, and a one-second total backoff
budget. When connection failures return quickly, one wake-up can invoke
`updateMember()` about twelve times.
2. Each invocation tries every configured PD URL. With three unreachable PD
endpoints, a single wake-up can therefore emit about 36 copies of the INFO
message before the batch-level warning is logged.
3. Membership checks are not triggered only by the one-minute periodic ticker.
A failed service-mode check schedules one every three seconds. Failed TSO,
region, store, keyspace, meta-storage, resource-manager, router, and HTTP
requests can also call `ScheduleCheckMemberChanged()`.
4. TSO reconnection is a particularly hot path: stream creation and request
failures schedule membership checks while TiDB continues requesting
timestamps for active SQL transactions.
5. `checkMembershipCh` has capacity one and coalesces concurrent notifications,
but it does not rate-limit sustained failures. A new notification can remain
queued while the current retry batch runs, causing another batch to start as
soon as the current one finishes.
Representative TSO call path:
```text
TiDB statement starts or activates a transaction
-> client-go requests a timestamp from the PD client
-> TSO stream creation or request fails
-> ScheduleCheckMemberChanged()
-> updateMemberLoop()
-> Backoffer.Exec(updateMember)
-> updateMember tries every PD URL
-> "[pd] cannot update member from this url"
```
Ordinary PD RPC failures follow a similar path through the common response error
handler. Multiple TiDB instances, keyspaces, and PD client instances multiply
the resulting request and log volume further.
### Design
Preserve the existing complete fast retry batch, then enter a degraded member
refresh mode only when every current member URL fails with a transport error and
every corresponding gRPC connection is observed in `IDLE`, `CONNECTING`, or
`TRANSIENT_FAILURE`.
Here, a transport error means an explicit gRPC dial failure or an RPC failure
classified as `Unavailable` or `DeadlineExceeded` (including a local context
deadline). This classification is only an eligibility signal; the
connection-state condition above is independently required.
While degraded:
- Coalesce asynchronous membership-check requests without issuing
`GetMembers`.
- Inspect local connection state every 100 ms and call `Connect()` for idle
connections.
- Resume the existing retry batch when a connection becomes `READY`, is
missing, is shut down, or the member URL set changes.
- Perform one real membership safety sweep on the existing one-minute tick.
- Keep synchronous `CheckMemberChanged` calls immediate and unsuppressed.
Track transport-failure episodes independently for each service-discovery client
and URL. Log the first detailed error, emit one-minute aggregate summaries for
suppressed errors, and log recovery for the URL that recovers.
### Expected behavior
- Control degraded-mode decisions with structured transport errors and gRPC
connectivity state; do not infer them from error-message text.
- Keep response-header errors, cluster-ID mismatches, application-level errors,
and uncertain failures on the existing retry path.
- Preserve the existing complete retry batch, timeouts, returned errors,
callbacks, and synchronous refresh behavior.
- Avoid additional healthy-mode tickers, background goroutines, configuration,
dependencies, or metrics.
- Reduce repeated asynchronous membership requests and detailed per-URL logs
during sustained transport outages without delaying state-driven recovery.
Contributor guide
Research direction
Start in service_discovery.go at updateMemberLoop() and updateMember(), then trace ScheduleCheckMemberChanged(), CheckMemberChanged(), checkMembershipCh, and Backoffer.Exec(updateMember). Verify transport-error classification and gRPC connectivity states against the stated degraded-mode rules; done means asynchronous failures are coalesced and summarized while synchronous refreshes, recovery, and the existing retry behavior remain intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100