microsoft / microsoft/pai

[CI] Setup nightly deployment to do the e2e tests and agile CI

Open
#5,173 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pai-dev
Dominant language
JavaScript
Stars
2.7k
Forks
554
Avg merge
20h 42m
Merged PRs (30d)
14

Description

Motivation

Current CI for pull request cost more than 30 minutes, we always have to wait for the CI for a long time, here is the details:

  1. Jenkins:
    1. Build images (1~2 min)
    2. Deploy (11~12 min)
    3. Run tests (10~11 min)
    4. Clean cluster (6~7 min)
  2. GitHub Actions:
    1. CI (4~5 min)
      1. Deployment test
      2. Exporter test
      3. Watchdog test
      4. Rest-server test
      5. Webportal test
      6. Webportal Plugin test
    2. Lint (<1 min)
  3. Travis-ci: (<5 min)

If we setup a nightly deployment/test and move some heavy part from CI to the it, we can agile CI and do more automatic tests.

CI-PR
jenkins-duration

Goal
  1. Agile CI in pull request
  2. The CI will contain unit tests and code coverages
  3. The nightly deployment/test will include PAI deploy and end-to-end tests
Proposal
  1. Move current Jenkins part from CI to the nightly deployment/test.
  2. Add build & unit-tests to GitHub Actions part to make sure the PR was buildable.
  3. Add code coverage threshold to CI.
  4. Add the heavy tests to nightly build.

In discuss meeting, add these items:

  1. Push nightly-build tag images to docker hub after nightly-build
  2. Add more feature to Lint action (swagger, CN document, …)
  3. Test for quick start deployment in nightly-deployment
  4. Manually trigger nightly build/tests
  5. Set GitHub action to require, finally move Travis-ci to GitHub Action

image

Discuss:

  1. Nightly-deployment/test trigger time
    a. 00:00

  2. How to alert when nightly build error
    a. Show in GitHub page
    b. Send email to commit owner

  3. code coverage threshold

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the existing Jenkins, GitHub Actions, and Travis-ci jobs first, using the current PR stages listed in the issue as the baseline. Define how nightly deployment/test will run PAI deploy and end-to-end tests while PR CI keeps unit tests and coverage. Done means the heavy tests move to the nightly workflow and the proposed CI goals are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions
Domain
ci-cd, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.