Smooth upgrade can leave TiDB in upgrading state after rollout completes

Bug Report

What version of Kubernetes are you using?

What version of TiDB Operator are you using?

v1.6.6

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

Not relevant to this issue.

What's the status of the TiDB cluster pods?

All three TiDB pods completed the rolling upgrade and were healthy. The TidbCluster reported Ready=True, phase: Normal, three updated replicas, and matching current/update revisions.

What did you do?

Upgraded TiDB from a v8.5.5 build to v8.5.7. After the component rollout completed, user DDL jobs remained paused with the ddl-upgrading category because TiDB was still in the global upgrading state.

Calling the following manually caused the paused DDL jobs to resume immediately:

curl -sS -X POST http://127.0.0.1:10080/upgrade/finish

Only post-recovery TidbCluster and StatefulSet YAML is available, so we cannot determine whether the smooth-upgrade annotations were missing before recovery or whether another finish condition remained unsatisfied.

What did you expect to see?

After all TiDB pods finish upgrading and become healthy, TiDB Operator should call /upgrade/finish and return the cluster to the normal state automatically. A lost or stale controller annotation should not leave user DDL paused indefinitely.

What did you see instead?

The Kubernetes rollout completed successfully, but TiDB remained in the upgrading state until /upgrade/finish was called manually.

The current reconciliation logic only attempts finish when tidb.pingcap.com/smooth-upgrade-ddl-paused=true is present. It does not compare that controller-owned state with TiDB's live /upgrade/show state. This creates a non-self-healing state mismatch if the annotation is lost or not observed.

Suggested improvement

Use /upgrade/show as a defensive reconciliation check while keeping the annotation as the primary workflow state:

• If the annotation is absent, TiDB reports upgrading, and the TiDB rollout is definitively complete, call /upgrade/finish and emit a warning event.
• If the annotation exists but TiDB already reports normal, clear the stale annotation.
• Do not automatically finish while the rollout is incomplete.
• Log or expose the reason when finish is waiting on replicas, phase, or member health.

This would make the smooth-upgrade workflow recover from state divergence even when the original trigger cannot be reconstructed.

Trace the smooth-upgrade reconciliation path that uses the smooth-upgrade annotation and the TiDB /upgrade/show and /upgrade/finish endpoints. Verify behavior for missing or stale annotations while rollout completion, replica status, phase, and member health are checked; done means divergent states recover safely without finishing an incomplete rollout, with the specified warning or waiting reason exposed.