tidb-operator unable to recover an unhealthy cluster even with manual revert

Bug Report

What version of Kubernetes are you using?

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-05-19T19:53:08Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

What version of TiDB Operator are you using?

pingcap/tidb-operator:v1.3.2

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

default

What's the status of the TiDB cluster pods?

NAME                                      READY   STATUS    RESTARTS   AGE     IP            NODE           NOMINATED NODE   READINESS GATES
test-cluster-discovery-779bb58fc7-q2kqv   1/1     Running   0          3m45s   10.244.4.11   kind-worker4   <none>           <none>
test-cluster-pd-0                         1/1     Running   0          3m45s   10.244.1.10   kind-worker3   <none>           <none>
test-cluster-pd-1                         1/1     Running   0          3m45s   10.244.4.12   kind-worker4   <none>           <none>
test-cluster-pd-2                         1/1     Running   0          3m45s   10.244.2.8    kind-worker    <none>           <none>
test-cluster-tidb-0                       2/2     Running   0          2m58s   10.244.2.9    kind-worker    <none>           <none>
test-cluster-tidb-1                       2/2     Running   0          2m58s   10.244.1.12   kind-worker3   <none>           <none>
test-cluster-tidb-2                       0/2     Pending   0          53s     <none>        <none>         <none>           <none>
test-cluster-tikv-0                       2/2     Running   0          3m31s   10.244.1.11   kind-worker3   <none>           <none>
test-cluster-tikv-1                       2/2     Running   0          3m31s   10.244.4.13   kind-worker4   <none>           <none>
test-cluster-tikv-2                       2/2     Running   0          3m31s   10.244.3.8    kind-worker2   <none>           <none>

What did you do?

We first have a healthy tidb cluster.
Then we changed the affinity rule for scheduling the tidb pods. The tidb statefulset is updated with the new affinity rule and performed a rolling upgrade. Then we realized that the new affinity rule cannot be satisfied for the current topology of the cluster, there is always one replica cannot be scheduled.
We then reverted the CR, to remove the new affinity rule.
But then the operator fails to update the tidb statefulset, leaving the cluster under unhealthy state.

What did you expect to see?
tidb-operator should be able to recover the cluster, with manual revert.

What did you see instead?
tidb-operator refuses to update the statefulset, before all replicas in the cluster become ready.
The log indicating this behavior is
tidb_cluster_controller.go:124] TidbCluster: default/test-cluster, still need sync: tidbcluster: [default/test-cluster]'s upgraded tidb pod: [test-cluster-tidb-2] is not ready, requeuing

Start by inspecting tidb_cluster_controller.go around line 124, then reproduce the affinity change and manual CR revert on Kubernetes. Trace why the controller requeues while tidb-2 is not ready. Done means the operator updates the TiDB StatefulSet after the revert and the cluster can recover to a healthy state.