pingcap / pingcap/tidb-operator

[v1] Rolling updates (scaleUp/Down) are processed one pod per reconciliation cycle

Open
#6,768 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

contribution
Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Summary

In v1, when upgrading TiDB pods (image/config/resource changes), the upgrader
processes exactly one pod per reconciliation cycle, regardless of how many
pods need updating.

Affected Component

pkg/manager/member/tidb_upgrader.go (and equivalent for TiKV, TiFlash, PD)

Root Cause

tidbUpgrader.Upgrade() iterates pod ordinals from highest to lowest and calls
upgradeTiDBPod() which sets the StatefulSet RollingUpdate.Partition to that
one ordinal. The reconciliation then returns and waits for that pod to become
available before proceeding to the next ordinal in the following cycle.

Impact

For N pods needing rolling updates, N reconciliation cycles are required — each
cycle introducing overhead from pod restart time, readiness probe wait, and
controller cache sync.

Related

Environment

  • Version: v1 (pkg/manager/member)
  • Applies to: TiDB, TiKV, TiFlash, PD upgraders

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pkg/manager/member/tidb_upgrader.go, tracing tidbUpgrader.Upgrade() and upgradeTiDBPod(), then compare the equivalent TiKV, TiFlash, and PD upgraders. Verify how StatefulSet RollingUpdate.Partition and reconciliation readiness interact. Done means eligible pods no longer require one reconciliation cycle each, with the behavior consistently addressed across the listed upgraders.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.