Proposal: Parallelize control-loops of same TiDB cluster

Feature Request

Is your feature request related to a problem? Please describe:
We've already parallelize our control-loops between different TiDB clusters, the workQueue ensures that different workers won't sync a same TidbCluster at the same time. However, for a specific TiDB cluster, all reconcile functions run in sequential, which increases risk of subsequent operations being blocked by failure or lag of one step. For example, it may takes a relatively long time to rolling-update TiKV, and the operator cannot perform failover for tidb-servers at that time because of the synchronization.

• A real case could be found here: https://github.com/pingcap/tidb-operator/pull/1242#discussion_r351140739)

Describe the feature you'd like:

Try to break tidb_cluster_controller into several sub-controllers that runs in parallel.

Teachability, Documentation, Adoption, Migration Strategy:

This is an notable change and it is safer to target it in v1.2.0

Start with the tidb_cluster_controller reconciliation flow and the workQueue behavior described in this issue; review the linked PR discussion for the real case. Map the reconcile functions that currently run sequentially and determine the safety boundaries for parallel sub-controllers. Done means independent control loops can run in parallel for one TiDB cluster without conflicting operations or blocking unrelated failover work.