pingcap / pingcap/tidb-operator
BR backup could raise error when PD leader changed during BR initialization
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 540
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 18
Description
Bug Report
What version of Kubernetes are you using?
What version of TiDB Operator are you using?
What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
What's the status of the TiDB cluster pods?
What did you do?
- Follow any official document to backup a cluster using CR (for example, https://github.com/pingcap/tidb-operator/blob/master/cmd/backup-manager/app/backup/backup.go#L237)
- During the BR initialization, switch PD leader to a different pod or offline PD leader
- BR job will raise following error:
error=\"pd address not available, ..., dial tcp: lookup <pd addr>: no such host, please check network - This is most likely due to when executing BR using operator, only the PD leader address is used to discover PD cluster memberlist. The TiUP BR allows to add multiple PD addresses in the command line to prevent one PD failure during the discovery, maybe operator should also consider this. Code ref: https://github.com/pingcap/tidb-operator/blob/master/cmd/backup-manager/app/backup/backup.go#L237
What did you expect to see?
BR is able to run when PD leader is offline during discovery
What did you see instead?
BR failed and raised an error
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in cmd/backup-manager/app/backup/backup.go around line 237, where the operator executes BR using the PD address. Reproduce or inspect the initialization flow with the PD leader changed or offline, and compare it with the reported multiple-address discovery behavior. Done means BR can complete discovery and run when the original PD leader is unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100