An enhanced rolling restart strategy based on the current implementation.

Feature Request

Is your feature request related to a problem? Please describe:

TiDB Operator uses the sts partition mechanism to implement fine-grained rolling updates,it still has some limitations that user can only restart pods in order from high to low according to the sts serial number.

In the past, we thought about some enhanced sts styles to enhance the control of pod. Refer:
#1094
#3287

These have not been effectively implemented and promoted, so i think we should refactor the existing implementation to really make the pod control more fine-grained.

Describe the feature you'd like:
We can implement strong pod control based on the onDelete update strategy of sts. STS only responsible for configuration update and deployment, we customize the implementation of the delete pod strategy. For example [1,2,3,4,5] five pods, we can restart [1,2] two pods , then restart [4,5,3] three pods.

We can develop a pre-selection mechanism to select pods that can be restarted first, rather than being limited to the order of sts.
For example, suppose we find that the instance data is damaged. After we manually offline this instance by pd-ctl tool, we need to transfer this pod to a new machine, then we should restart this pod first.
Based on this, we can have more considerations, such as traffic, such as the CPU memory of the node.

Describe alternatives you've considered:
We can completely migrate the current rolling update logic based on onDelete strategy , but after we implement this set of pre-selection mechanism, we can have a more flexible rolling restart strategy framework.

Start by reading the existing StatefulSet rolling-update logic and the onDelete strategy, then review issues #1094 and #3287 for prior approaches. Define how pod pre-selection should support arbitrary restart order and account for manually offlined instances, traffic, and node CPU or memory before implementing the strategy.