pingcap / pingcap/tidb-operator

Operator cannot upgrade version when some Pods crash

Open
#6,155 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Bug Report

What version of Kubernetes are you using?
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.4-12.8d683d9", GitCommit:"8d683d982b20a8f28a62ad502db0f352e50f621c", GitTreeState:"clean", BuildDate:"2019-12-30T09:24:27Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.16) exceeds the supported minor version skew of +/-1

What version of TiDB Operator are you using?
TiDB Operator Version: version.Info{GitVersion:"v1.4.7-1+1200b7c2d69962", GitCommit:"1200b7c2d69962a28079109a2999aa17eb7f6ec6", GitTreeState:"clean", BuildDate:"2025-02-12T08:50:37Z", GoVersion:"go1.23.5", Compiler:"gc", Platform:"linux/amd64"}

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

What's the status of the TiDB cluster pods?

CrashLoopBackOff
What did you do?

  1. Upgrade tikv from tikv 4.0.8 to 5.4.0
  2. Before all node restart, scale in tikv replica from 33 to 32(there is still have v4.0.8 pod)
  3. Upgrade blocked by scale in
  4. Kill tikv-25( tikv-25 is v5.4.0 for now)
  5. After tikv-25 start, it become v4.0.8 and run into crash
  6. edit tc ,change tikv`s storage from 1500GB to 2000GB
  7. After scale in completed, upgrade begin from tikv-31
  8. When evict-leader tikv-25, as tikv-25 is in crash ,operator can`t get how many leaders in tikv-25. operator will enter dead loop for get leader count for tikv-25
  9. What did you expect to see?
    upgrade will continue
    What did you see instead?
    upgrade blocked

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the TiKV upgrade flow and the leader-count handling used during eviction, then reproduce the reported sequence with a CrashLoopBackOff TiKV pod. Confirm the operator does not loop indefinitely when leader information cannot be retrieved and that the upgrade proceeds afterward; the payload names no source files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.