h2oai / h2oai/datatable

[CI] Restartable pipeline

Open
#2,267 6 comments 0 reactions 3 assignees Claimed by @achraf-mer View on GitHub
improve ITA
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

Currently, the Jenkins pipeline is organized as a sequence of steps such that whenever one of the steps fail, the pipeline is terminated, and then has to be restarted from the very beginning once the issue is addressed. This is problematic, especially if the steps before the failure have been successful, and re-running them will waste significant amount of time.

This proposal suggests to re-organize the pipeline around the idea that each step in the pipeline should persist its result in S3, and then if restarted, Jenkins should skip those stages that have been successful before.

The suggested structure is the following:
1. First, the master task performs git checkout, and immediately builds an `sdist` of the datatable. Building `sdist` requires only standard python (3.5+), and can be done either in a docker, or in a virtual env, or without any virtualization at all. The `sdist` produces a single `.tar.gz` file, which contains all files that are considered "source" of the datatable. This sdist is stored in S3.
2. Similarly, we tarball all tests in the datatable repo, and also stash them in S3.
3. The build stage(s) produce binary wheels of the datatable based on the `sdist` file. The output wheels are stored in S3, tagged with the sha-hash of the sdist. The stage can be skipped if the wheels already exist.
4. Each test stage grabs the wheel file, and the tests-tarball from S3; runs the tests; and stores the "success" marker on S3 (which could be a 0-size file). The stage can be skipped if the corresponding marker (tagged with sha of the wheel and the tests-tarball) already exists in S3.
5. The final stage verifies the existence of all wheels/test results in S3, and performs any epilogue actions.

This structure ensures that:
- if the source code changes, the entire pipeline is rebuilt (same as now);
- if only the tests change, then all tests are re-run but wheels are not rebuilt;
- if only auxiliary files change (docs, jenkins scripts, git support files, etc), or no files change at all (e.g. Jenkins job restart), then only steps that failed before need to rerun.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.