improve the revision check in upgrader

Bug Report

What version of Kubernetes are you using?
v1.19

What version of TiDB Operator are you using?
from master

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

What's the status of the TiDB cluster pods?

What did you do?
this is not exactly a tidb-operator issue and the root cause could be in k8s, but enhance tidb-operator could workaround the issue

during tidb deployment using tidb-operator, we found the process is stuck, with the status like

jq ".status.tikv.statefulSet"
{
  "collisionCount": 0,
  "currentReplicas": 21,
  "currentRevision": "mussel-prod-tikv-7cb857cc78",
  "observedGeneration": 1,
  "readyReplicas": 30,
  "replicas": 30,
  "updateRevision": "mussel-prod-tikv-7cb857cc78",
  "updatedReplicas": 21
}

the status of tikv pods looks like

mussel-prod-tikv-0                                  3/3     Running       0          5d22h
mussel-prod-tikv-1                                  3/3     Running       0          6d20h
mussel-prod-tikv-10                                 3/3     Running       0          15d
mussel-prod-tikv-11                                 3/3     Running       0          6d20h
mussel-prod-tikv-12                                 3/3     Running       0          7d19h
mussel-prod-tikv-13                                 3/3     Running       0          5d22h
mussel-prod-tikv-14                                 3/3     Running       0          12d
mussel-prod-tikv-15                                 3/3     Running       0          12d
mussel-prod-tikv-16                                 3/3     Running       0          5d19h
mussel-prod-tikv-17                                 3/3     Running       0          6d19h
mussel-prod-tikv-18                                 3/3     Running       0          5d22h
mussel-prod-tikv-19                                 3/3     Running       0          15d
mussel-prod-tikv-2                                  3/3     Running       0          12d
mussel-prod-tikv-20                                 3/3     Running       0          5d22h
mussel-prod-tikv-21                                 3/3     Running       0          5d20h
mussel-prod-tikv-22                                 3/3     Running       0          12d
mussel-prod-tikv-23                                 3/3     Running       0          5d20h
mussel-prod-tikv-24                                 3/3     Running       0          5d19h
mussel-prod-tikv-25                                 3/3     Running       0          5d21h
mussel-prod-tikv-26                                 3/3     Running       0          5d20h
mussel-prod-tikv-27                                 3/3     Running       0          5d21h
mussel-prod-tikv-28                                 3/3     Running       0          5d20h
mussel-prod-tikv-29                                 3/3     Running       0          5d20h
mussel-prod-tikv-3                                  3/3     Running       0          7d19h
mussel-prod-tikv-4                                  3/3     Running       0          7d19h
mussel-prod-tikv-5                                  3/3     Running       0          12d
mussel-prod-tikv-6                                  3/3     Running       0          6d19h
mussel-prod-tikv-7                                  3/3     Running       0          15d
mussel-prod-tikv-8                                  3/3     Running       0          6d20h
mussel-prod-tikv-9                                  3/3     Running       0          12d

all the tikv running for longer than 10d are not on the latest revision, and the old revision one are not in order.

this is weird and likely some race condition in k8s but not tidb-operator, and we are not sure how we got into this situation
but once this happens, tidb-operator stuck to make process. dig into the code, the operator checks the status of every pod https://github.com/pingcap/tidb-operator/blob/master/pkg/manager/member/tikv_member_manager.go#L992, so it decides there should be a upgrade in this case, but skips the upgrade later because the currentRevision equals to updateRevision https://github.com/pingcap/tidb-operator/blob/master/pkg/manager/member/tikv_upgrader.go#L105

so i would suggest to check if currentReplicas equals replicas in the line to check currentRevision as well, what do you think? if this sounds fine, i could go ahead and add it

What did you expect to see?
operator could make progress

What did you see instead?
operator stuck

Start in pkg/manager/member/tikv_member_manager.go at the pod status check and pkg/manager/member/tikv_upgrader.go at the currentRevision check. Compare the revision and replica conditions against the reported StatefulSet status. Done means the upgrader does not skip a needed upgrade and the TiDB deployment can make progress.