Improve e2e test framework

Feature Request

Is your feature request related to a problem? Please describe:

Currently the e2e cases are flaky and hard to debug, which hinders the development process like PR merge.
We should Make E2E Test Great Again (MEGA)!

Describe the feature you'd like:

A stable, debuggable, easy to write, understandable e2e test framework with no surprises.

Describe alternatives you've considered:

None.

Teachability, Documentation, Adoption, Migration Strategy:

Things needs to be done in small PRs as possible.
We can reference to Writing good e2e tests for Kubernetes

Here is the proposed todo list:

• P1
  • Summarize E2E cases matrix @DanielZhangQD
  • break down test cases into smallest units @dragonly
    • #3729
  • unify logging interface @dragonly
    • #3639
  • failures must have reasons (e.g. calls to framework.ExpectNoError() must have explain parameter) @dragonly
    • #3663
  • List E2E test cases according to the scenarios that need to be covered
    • TidbCluster controller
  • Refactor the existing E2E cases, clean up unreasonable code, ensure the stability of the current PR test
    • Remove unnecessary Helm Chart deployment @handlerww
    • Remove unnecessary LocalPV support @handlerww
• P2
  • Modify or add new cases according to the case list (in accordance with the E2E code pattern and description)
    • TiDBCluster
      • upgrade tc version #3775 @dragonly
      • delete resources controlled by tc #3798 @dragonly
      • scale
      • failover
    • TiDBMonitor
      • deploy
      • migration
    • TiDBInitializer
      • InitSql, InitSqlConfigMap, Password, PermitHost
    • DMCluster
      • Back-forward compatibility
    • TiDB Scheduler
      • prevent skew deployment
    • Backup
      • backup to different provider
      • export to different provider
      • scheduler backup
      • restore
      • import
    • webhook
      • StatefulSet validating webhook
      • CR validating webhook
      • CR mutating webhook
      • Pod webhook (To be removed in #3497)
    • Lightning
      • local-backend for TLS, non-TLS
    • Drainer
      • sync to file
  • Check whether the incorrect CM issue where TLS configurations are added to the non-TLS clusters still exists
  • Deliver the E2E test specification document and add it to the repo (add a link in contribution.md)
  • List E2E testcases according to the scenarios that need to be covered
    • Other controllers
• P3
  • Group the cases and decide when to run the cases (run for each PR or run periodically)
    • e2e test cases should be split into different groups (e.g. normal quick tests, [Slow], [Serial], [Feature:.*])
    • provide parallel E2E tests on Jenkins @handlerww
  • UT specification
• P4
  • Refactoring and optimization of Stability cases
  • GKE, EKS test

Start by reviewing the proposed E2E todo list and the existing E2E cases for the affected controller or feature area. Use the listed case matrix and E2E code pattern as the scope, and consult contribution.md for the planned specification link. Done should be a small, independently reviewable improvement with stable, understandable tests and documented behavior.