When upgrad or reload a cluster, increase retry when accessing PD
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 466
- Forks
- 338
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 8
Description
Bug Report
- What did you do?
tiup cluster upgrade <clsuter_name>
In the TiKV evict leader phase :
error requesting pd api , response: no leader
- What did you expect to see?
After investigation, it was found that due to the leader priority setting in PD, a leader switch occurred during the "upgrade cluster" pd stage. Subsequently, PD checked the leader priority every minute, causing a PD leader transfer that took 0.5 seconds.
Coincidentally, during this 0.5-second window, the upgrade cluster process had already reached the TiKV stage and was performing the "set leader evict scheduler" operation, resulting in a "no leader" error when accessing PD, which caused TiUP to exit.
I think a retry mechanism should be added when calling the PD API to prevent TiUP upgrade or reload operations from being interrupted due to such short-term changes in PD.
-
What did you see instead?
tiup error exits -
What version of TiUP are you using (
tiup --version)?
v1.14.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the TiKV evict-leader phase and the PD API call used by the cluster upgrade and reload paths. Reproduce or inspect handling of the transient "no leader" response, then verify that a short PD leader transfer no longer causes the operation to exit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100