ci: Refactor and optimize the CI pipelines

Enhancement

The current pipelines has some issues, which have brought a significant burden to the maintenance and updates of the pipelines.

• testing logic and environment preparation logic are mixed in the Groovy file
• There is a triggering relationship between the upstream and downstream calls of the pipelines.

The plan is to restructure CI Pipelines through intentional redesign, making the structure of the pipelins simpler and its execution more stable.

• Refactor the script-based Jenkins Groovy to declarative Groovy.
• The environment preparation script(like prepare cache and tools) is implemented using shell script instead of Groovy script.
• Using Prow to take over the triggering of pipelines.
• Remove the build-common CI pipeline and directly handle related tasks in the unit-test and integration-test pipelines.
• Each branch has an independent pipeline configuration to avoid adding conditional logic in the pipeline that is compatible with the differences of multiple branches.

Which CI piplines will be affected?

• https://ci.pingcap.net/view/tiflash/job/tiflash-build-common/
• https://ci.pingcap.net/view/tiflash/job/tiflash-ghpr-build/
• https://ci.pingcap.net/view/tiflash/job/tiflash-ghpr-integration-tests/
• https://ci.pingcap.net/view/tiflash/job/tiflash-ghpr-unit-tests/

New CI pipelines

• https://do.pingcap.net/jenkins/job/pingcap/job/tiflash/

Which branches' pipelines will be affected?

This optimization only targets the pipeline of the master branch. The remaining release branches will be switched and updated in the coming weeks.

Which behaviors will change

After this adjustment, the pipelines will be reduced from three to two. The functionality steps of tiflash_ghpr_build will be integrated into pull-integration-tests as a whole.

old pipelines

• tiflash-ghpr-unit-tests
• tiflash-ghpr-integration-tests
• tiflash-ghpr-build

new pipelines

• pull_unit_test
• pull_integration_test

1. The CI trigger command will change
   /run-unit-test --> /test pull-unit-test
/run-integration-test --> /test pull-integration-test
/run-all-tests --> /test all

2. CI status will change
   idc-jenkins-ci-tiflash/integration-test --> pull-integration-test
idc-jenkins-ci-tiflash/unit-test --> pull-unit-test

3. Automatic triggering of the pielines
   For newly created PRs or PRs with new commits pushed, the pipeline will be triggered automatically. No need to comment /test all to trigger manually. Both pull-unit-test and pull-integration-test will be triggered automatically.

4. Prefer to use /retest to trigger tasks that have failed, /test all will retrigger all tasks (including those that have run successfully and those that have failed).

How to check my pull requst ci status on prow

https://prow.tidb.net/?repo=pingcap%2Ftiflash&type=presubmit

Any others changes I need to pay attention to?

• This refactoring optimization only targets the master branch, currently other branches are not affected.
• The PR batch merge will be supported in the future.

Start by reviewing the four affected Jenkins pipelines linked in the issue and the new pingcap/tiflash pipeline configuration for the master branch. Compare the current three-pipeline flow with the proposed pull_unit_test and pull_integration_test flow, including triggers and status names; done means the redesigned pipelines run stably with the documented automatic triggers and behavior.