oasisprotocol / oasisprotocol/oasis-core

Fix the CI situation

Open
#4,035 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

c:build c:testing p:0
Dominant language
Go
Stars
369
Forks
151
Avg merge
1d 3h
Merged PRs (30d)
6

Description

The current CI situation is beyond awful, with time to completion easily exceeding 1 hour. This makes doing development excruciatingly painful (despite what certain people seem to think), so effort needs to be spent in resolving this.

Root causes:

  • SGX buildkite agents are extremely limited. Having to wait for over 30 mins just for an agent to become available so tests can start is the rule rather than the exception.
  • Automated PRs add to agent starvation. While it is nice to have things like dependabot, what is not so nice is something that spams PRs that in turn generate CI runs (that will never complete successfully in a mergable state due to the github action checks failing 100% of the time).
  • Build times are excessive, though I am told this is part of what makes Rust a modern and amazing language.
  • Test execution times are excessive. We have a lot of tests, most which require spinning up a new network instance, and expensive epoch transitions.

While I'm complaining about CI:

  • It is the Year of Our Lord 2021. It is idiotic that the changelog fragment format is so stringent. Builds failing because of a trailing newline should not be a thing. And expecting developers to install the burning trashfire that is node.js and npm to avoid this is unrealistic.
  • I for one try (or tried) to put effort into my github commit messages, why are changelog fragments a thing? A non-trivial part of my changelog fragment misery is because I'm copy/pasting from git log into a text file by hand (Read this as: Why isn't this automated?).
  • Having to create a PR to get a number to placate the changelog fragment linter, which kicks off CI, only to push a fragment (for trivial PRs) is wasteful.

Remedies (in no particular order):

  • PRs that trigger an automated CI cycle should exclude dependabot, and perhaps flat out be limited by user.
  • dependabot PRs should be mergable "as is" once they have gone through CI, and not "after someone adds one of those stupid fragment things", which requires(?) another CI cycle. (#4091)
  • The Rust runtime client(s) used in tests should be rewritten in Go, to reduce build time. (#4038)
  • More parts of the buildkite CI pipeline should be moved over to github actions (#3517).
  • Our tests need to be re-architected so that they run faster. The problem got notably worse when I unified timekeeping (so perhaps making mock epoch transitions faster somehow is sufficient), but repeatedly spinning up/tearing down the network is not great.
  • Consider reducing the number of tests that require SGX, or moving some of the "expected to pass" SGX tests to only run on merge.
  • All use of grpcio should be removed from the codebase, and future use of the crate should be banned.
  • "F*** it, we'll throw money at the problem". Add more agents.
  • ci-lint should probably be skipped for draft PRs for frustration reduction.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Buildkite CI pipeline, the test execution setup, and ci-lint behavior for draft PRs; the issue names no specific files or tests. Review the referenced issues #3517, #4038, and #4091 to separate completed work from the remaining items. Done would require a scoped, measurable reduction in CI waiting or execution time.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, rust
Domain
build-system, ci-cd, devops, testing-qa
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.