mcs: silent store heartbeat forwarding failure causes false down-store alerts
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
Deployed the scheduling service independently. During a PD leader switch, the `servicePrimaryMap` in PD temporarily loses the primary address of the scheduling service until the etcd watcher catches up. During this window, `GetServicePrimaryAddr` returns an empty string, causing the scheduling gRPC client to evaluate to `nil` in `updateSchedulingClient`.
When `StoreHeartbeat` requests arrive at PD, they are handled locally and then forwarded. However, when the scheduling client is `nil`, the forwarded heartbeat is silently dropped at `server/grpc_service.go:954-972`.
### What did you expect to see?
Forwarding failures should be observable via metrics, just like `RegionHeartbeat` forwarding failures, so that the silent dropping of heartbeats doesn't go unnoticed.
### What did you see instead?
The failure is entirely silent. No metrics are incremented, and no errors are logged or returned. Because the scheduling service starves for `StoreHeartbeat`s, the stores' `LastHeartbeatTS` age out locally on the scheduling server, eventually causing the stores to transition from `disconnected` -> `unhealthy` -> `down`. This triggers a false `SchedulingServerDiscoverDownStore` alert, even though the stores are perfectly healthy and successfully heartbeating to PD.
### What version of PD are you using (`pd-server -V`)?
v8.5.4-nextgen.202510.7
Contributor guide
Assessment
This issue has not been assessed yet.