TiFlash stays disconnected for minutes after PD leader transfer due to proxy pd_client NO_RETRY

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Deploy a TiDB cluster with PD (multi-member) and TiFlash.

2. Confirm TiFlash stores are healthy / connected on the PD side.

3. Manually transfer PD leader, for example:

   # From any PD endpoint, transfer leadership to the target PD member
   curl -X POST http://<pd-addr>:2379/pd/api/v1/leader/transfer/<new-pd-name>
   # or via pd-ctl equivalent
   

4. Observe PD store status for TiFlash and TiFlash proxy (tiflash_tikv) logs around the transfer.

Observed incident timeline (2026-07-25, op gogame cluster; PD leader transfer from pd-10.107.16.121-2379 to pd-10.107.15.193-2379):

2. What did you expect to see? (Required)

After PD leader transfer:

• TiFlash proxy PD client should reconnect to the new PD leader promptly (similar to TiKV).
• store_heartbeat / report_min_resolved_ts should succeed against the new leader within a short window (seconds).
• PD should not report TiFlash stores as disconnected for minutes.

3. What did you see instead (Required)

• PD shows TiFlash stores as disconnected for ~3.5 minutes after leader transfer.

• TiFlash compute C++ PD client switches leader quickly (switch leader ..., source=pingcap.pd).

• TiFlash proxy (Rust pd_client) keeps talking to the old leader and repeatedly logs:

  report min resolved_ts failed  err=Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", ... }))
  store heartbeat failed         err=Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", ... }))
  

Root cause (code):

In contrib/tiflash-proxy/components/pd_client/src/client.rs, periodic RPCs use NO_RETRY = 1:

const LEADER_CHANGE_RETRY: usize = 10;
// periodic request like store_heartbeat, we don't need to retry.
const NO_RETRY: usize = 1;

Both store_heartbeat and report_min_resolved_ts call:

self.pd_client.request(req, executor, NO_RETRY).execute()

Even though gRPC not leader is retryable (Error::Grpc), with NO_RETRY the request returns immediately and never reaches reconnect_if_needed() → reconnect(true). Upper layers in raftstore only log the error and do not force reconnect.

Unlike TiKV, TiFlash is a Learner and typically does not send region heartbeats as leader, so it cannot use the region-heartbeat stream-disconnect path that forces TiKV to reconnect quickly. Recovery then depends on slower side paths (periodic update_loop with default update_interval = 10m, and/or TSO stream close → reconnect(true)), which matched the ~3.5 minute delay in this incident.

4. What is your TiFlash version? (Required)

• v8.5.4
• Component under investigation: contrib/tiflash-proxy PD client (components/pd_client)

Start in contrib/tiflash-proxy/components/pd_client/src/client.rs, focusing on store_heartbeat, report_min_resolved_ts, NO_RETRY, and reconnect_if_needed(). Trace how a gRPC "not leader" error is handled after PD leader transfer. Done means the proxy reconnects promptly and both periodic RPCs succeed against the new leader instead of leaving TiFlash disconnected for minutes.