tikv / tikv/pd

PD unable to recover when TSO primary deleted

Open
#11,035 1 comment 0 reactions 0 assignees View on GitHub
contribution first-time-contributor type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

In PD microservices mode the pd (API) server forwards every client TSO request to the pd_tso primary, resolving the primary's address from an in-memory cache (servicePrimaryMap[tso]) maintained solely by the tsoPrimaryWatcher (an etcdutil.LoopWatcher). The read path, GetServicePrimaryAddr, only reads that cache; the self-heal ForceLoad() is triggered only on IsLeaderChanged stream errors, never when the cached address is simply absent.

If the watcher ever misses the PUT of a new primary — e.g. an etcd watch gap during an etcd-leader stall, where the watch delivers the DELETE of a primary key but not the subsequent PUT — servicePrimaryMap[tso] is left empty and never repopulated on the request path. getGlobalTSO and handleTSOForwarding then return ErrNotFoundTSOAddr (gRPC NotFound "not found tso address") to every client indefinitely, until the PD process is restarted (which re-runs initFromEtcd → load).

### What did you do?

PD node's servicePrimaryMap[tso] was emptied by a delete with no following put and stayed empty. TiKV TSO workers got persistent NotFound "not found tso address", could not update max_ts (MaxTimestampNotSynced), and the SQL layer fell into a regionMiss/replica-selector retry storm at ~50% throughput. Restarting that single PD node cleared it instantly.

### What did you expect to see?
PD should be able to find the right TSO after re-election

### What did you see instead?
PD stuck with no TSO address even though TSO was elected.

### What version of PD are you using (`pd-server -V`)?
8.5.5

Contributor guide

Open the contributing guide

Research direction

Trace GetServicePrimaryAddr, servicePrimaryMap, and the tsoPrimaryWatcher's ForceLoad behavior, then follow the getGlobalTSO and handleTSOForwarding request paths. Compare this with initFromEtcd loading and the described DELETE/PUT watch sequence; done means a deleted or missing cached primary can recover after re-election without restarting PD and no longer returns ErrNotFoundTSOAddr.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.