pingcap / pingcap/tidb-operator

TiKVWorkerGroup version upgrade is blocked by unknown component in default upgrade policy

Open
#6,990 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Bug Report

What happened

When a TiKVWorkerGroup performs a version upgrade under the default cluster upgrade policy, the controller rejects the upgrade precondition check with:

failed to check preconditions for upgrading
component: tikv-worker
error: unknown component: tikv-worker
result: false

The group remains Ready=True but not fully synced/up-to-date because the upgrade checker never allows the version rollout to proceed.

Why this happens

TiKVWorkerGroup uses the generic upgrade checker in pkg/controllers/tikvworkergroup/tasks/updater.go:

checker := action.NewUpgradeChecker[scope.TiKVWorkerGroup](c, state.Cluster(), logger)
if needVersionUpgrade(obj) && !checker.CanUpgrade(ctx, obj) {
    ...
}

needVersionUpgrade becomes true when the desired template version differs from the observed status version:

wg.Spec.Template.Spec.Version != wg.Status.Version && wg.Status.Version != ""

For scope.TiKVWorkerGroup, the generated component name is tikv-worker:

func (TiKVWorkerGroup) Component() string {
    return v1alpha1.LabelValComponentTiKVWorker
}

But the default upgrade policy switch in pkg/action/upgrader.go does not handle tikv-worker, so it falls into the default branch:

return false, fmt.Errorf("unknown component: %s", scope.Component[S]())
Expected behavior

A supported TiDB Operator component type should not fail version upgrade precondition checking with unknown component.

TiKVWorkerGroup version upgrades should either:

  1. have an explicit default-policy rule, or
  2. intentionally share the same dependency/precondition behavior as TiKV, or
  3. be explicitly treated as no-dependency/always-allowed if that is the intended semantics.
Why this is surfacing now

Previously some deployment systems only updated spec.template.spec.image for hotfix/image-tag rollouts. Image-only changes do not satisfy needVersionUpgrade, so they bypass this upgrade checker.

Now deployment systems may update both:

spec.template.spec.image
spec.template.spec.version

Once spec.template.spec.version changes, TiKVWorkerGroup enters the generic upgrade checker and hits the missing tikv-worker case.

Impact

Any TiKVWorkerGroup version upgrade with Cluster.spec.upgradePolicy: Default can get stuck before updating instances.

A temporary workaround is setting:

spec:
  upgradePolicy: NoConstraints

but that bypasses normal default upgrade ordering/health checks, so it is not ideal as a long-term fix.

Suggested fix

Add tikv-worker support to the default upgrade checker in pkg/action/upgrader.go.

It may also be worth auditing other generated component names that can enter the same generic checker but are not handled by the default policy switch, for example router and resource-manager.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/controllers/tikvworkergroup/tasks/updater.go to trace when version upgrades invoke the checker, then inspect the default-policy switch in pkg/action/upgrader.go and the TiKVWorkerGroup Component method. Compare the existing TiKV behavior with the intended dependency semantics for tikv-worker, and check whether router or resource-manager follow the same path. Done means a Default-policy version upgrade no longer fails with an unknown component.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.