pingcap / pingcap/tidb-operator

BR backup could raise error when PD leader changed during BR initialization

Open
#5,630 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Bug Report

What version of Kubernetes are you using?

What version of TiDB Operator are you using?

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

What's the status of the TiDB cluster pods?

What did you do?

  1. Follow any official document to backup a cluster using CR (for example, https://github.com/pingcap/tidb-operator/blob/master/cmd/backup-manager/app/backup/backup.go#L237)
  2. During the BR initialization, switch PD leader to a different pod or offline PD leader
  3. BR job will raise following error:
    error=\"pd address not available, ..., dial tcp: lookup <pd addr>: no such host, please check network
  4. This is most likely due to when executing BR using operator, only the PD leader address is used to discover PD cluster memberlist. The TiUP BR allows to add multiple PD addresses in the command line to prevent one PD failure during the discovery, maybe operator should also consider this. Code ref: https://github.com/pingcap/tidb-operator/blob/master/cmd/backup-manager/app/backup/backup.go#L237

What did you expect to see?
BR is able to run when PD leader is offline during discovery

What did you see instead?
BR failed and raised an error

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in cmd/backup-manager/app/backup/backup.go around line 237, where the operator executes BR using the PD address. Reproduce or inspect the initialization flow with the PD leader changed or offline, and compare it with the reported multiple-address discovery behavior. Done means BR can complete discovery and run when the original PD leader is unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.