tikv / tikv/pd

mcs: silent store heartbeat forwarding failure causes false down-store alerts

Open
#10,619 1 comment 0 reactions 0 assignees View on GitHub
contribution type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

Deployed the scheduling service independently. During a PD leader switch, the `servicePrimaryMap` in PD temporarily loses the primary address of the scheduling service until the etcd watcher catches up. During this window, `GetServicePrimaryAddr` returns an empty string, causing the scheduling gRPC client to evaluate to `nil` in `updateSchedulingClient`.

When `StoreHeartbeat` requests arrive at PD, they are handled locally and then forwarded. However, when the scheduling client is `nil`, the forwarded heartbeat is silently dropped at `server/grpc_service.go:954-972`.

### What did you expect to see?

Forwarding failures should be observable via metrics, just like `RegionHeartbeat` forwarding failures, so that the silent dropping of heartbeats doesn't go unnoticed.

### What did you see instead?

The failure is entirely silent. No metrics are incremented, and no errors are logged or returned. Because the scheduling service starves for `StoreHeartbeat`s, the stores' `LastHeartbeatTS` age out locally on the scheduling server, eventually causing the stores to transition from `disconnected` -> `unhealthy` -> `down`. This triggers a false `SchedulingServerDiscoverDownStore` alert, even though the stores are perfectly healthy and successfully heartbeating to PD.

### What version of PD are you using (`pd-server -V`)?

v8.5.4-nextgen.202510.7

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.