pingcap / pingcap/tidb-operator
Add leader transfer protection for manually deleted PD pods
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 540
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 18
Description
Problem Description
Currently, when users manually delete a PD pod, the TiDB Operator does not perform leader transfer before the pod is terminated. This creates a potential availability issue.
Current Behavior
- Operator-initiated deletions (during rolling updates, configuration changes): ✅ Have leader transfer protection via
preDeleteCheck() - User-initiated deletions: ❌ No protection, relies only on PD cluster's internal raft failover
Code Analysis
In pkg/controllers/pd/tasks/pod.go, the leader transfer logic is only executed when the operator actively decides to delete a pod:
if !reloadable.CheckPDPod(state.PD(), pod) {
// Only triggered for operator-initiated updates
if state.IsHealthy() || statefulset.IsPodReady(pod) {
wait, err := preDeleteCheck(...) // Leader transfer happens here
// ...
}
}
When users manually delete a pod, the operator only sees pod == nil and directly rebuilds without leader transfer.
Expected Behavior
When a user manually deletes a PD pod that is currently the leader, the operator should:
- Detect the manual deletion (pod in Terminating state)
- Check if the PD is the current leader
- Transfer leadership to a healthy peer before allowing deletion to proceed
- Rebuild the pod after deletion completes
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/controllers/pd/tasks/pod.go and trace the pod == nil and Terminating-state handling alongside preDeleteCheck(). Confirm how a manually deleted PD pod is detected and how leadership is checked and transferred before deletion completes. Done means a manually deleted leader pod transfers leadership to a healthy peer before rebuilding, while operator-initiated deletion behavior remains protected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100