pingcap / pingcap/tidb-operator
Operator cannot upgrade version when some Pods crash
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 540
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 18
Description
Bug Report
What version of Kubernetes are you using?
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.4-12.8d683d9", GitCommit:"8d683d982b20a8f28a62ad502db0f352e50f621c", GitTreeState:"clean", BuildDate:"2019-12-30T09:24:27Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.16) exceeds the supported minor version skew of +/-1
What version of TiDB Operator are you using?
TiDB Operator Version: version.Info{GitVersion:"v1.4.7-1+1200b7c2d69962", GitCommit:"1200b7c2d69962a28079109a2999aa17eb7f6ec6", GitTreeState:"clean", BuildDate:"2025-02-12T08:50:37Z", GoVersion:"go1.23.5", Compiler:"gc", Platform:"linux/amd64"}
What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
What's the status of the TiDB cluster pods?
CrashLoopBackOff
What did you do?
- Upgrade tikv from tikv 4.0.8 to 5.4.0
- Before all node restart, scale in tikv replica from 33 to 32(there is still have v4.0.8 pod)
- Upgrade blocked by scale in
- Kill tikv-25( tikv-25 is v5.4.0 for now)
- After tikv-25 start, it become v4.0.8 and run into crash
- edit tc ,change tikv`s storage from 1500GB to 2000GB
- After scale in completed, upgrade begin from tikv-31
- When evict-leader tikv-25, as tikv-25 is in crash ,operator can`t get how many leaders in tikv-25. operator will enter dead loop for get leader count for tikv-25
- What did you expect to see?
upgrade will continue
What did you see instead?
upgrade blocked
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the TiKV upgrade flow and the leader-count handling used during eviction, then reproduce the reported sequence with a CrashLoopBackOff TiKV pod. Confirm the operator does not loop indefinitely when leader information cannot be retrieved and that the upgrade proceeds afterward; the payload names no source files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100