pingcap / pingcap/tidb-operator

Add leader transfer protection for manually deleted PD pods

Open
#6,405 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement v2
Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Problem Description

Currently, when users manually delete a PD pod, the TiDB Operator does not perform leader transfer before the pod is terminated. This creates a potential availability issue.

Current Behavior

  • Operator-initiated deletions (during rolling updates, configuration changes): ✅ Have leader transfer protection via preDeleteCheck()
  • User-initiated deletions: ❌ No protection, relies only on PD cluster's internal raft failover

Code Analysis

In pkg/controllers/pd/tasks/pod.go, the leader transfer logic is only executed when the operator actively decides to delete a pod:

if !reloadable.CheckPDPod(state.PD(), pod) {
    // Only triggered for operator-initiated updates
    if state.IsHealthy() || statefulset.IsPodReady(pod) {
        wait, err := preDeleteCheck(...) // Leader transfer happens here
        // ...
    }
}

When users manually delete a pod, the operator only sees pod == nil and directly rebuilds without leader transfer.

Expected Behavior

When a user manually deletes a PD pod that is currently the leader, the operator should:

  1. Detect the manual deletion (pod in Terminating state)
  2. Check if the PD is the current leader
  3. Transfer leadership to a healthy peer before allowing deletion to proceed
  4. Rebuild the pod after deletion completes

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/controllers/pd/tasks/pod.go and trace the pod == nil and Terminating-state handling alongside preDeleteCheck(). Confirm how a manually deleted PD pod is detected and how leadership is checked and transferred before deletion completes. Done means a manually deleted leader pod transfers leadership to a healthy peer before rebuilding, while operator-initiated deletion behavior remains protected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.