oasisprotocol / oasisprotocol/oasis-core
Fix the CI situation
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 369
- Forks
- 151
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
The current CI situation is beyond awful, with time to completion easily exceeding 1 hour. This makes doing development excruciatingly painful (despite what certain people seem to think), so effort needs to be spent in resolving this.
Root causes:
- SGX buildkite agents are extremely limited. Having to wait for over 30 mins just for an agent to become available so tests can start is the rule rather than the exception.
- Automated PRs add to agent starvation. While it is nice to have things like dependabot, what is not so nice is something that spams PRs that in turn generate CI runs (that will never complete successfully in a mergable state due to the github action checks failing 100% of the time).
- Build times are excessive, though I am told this is part of what makes Rust a modern and amazing language.
- Test execution times are excessive. We have a lot of tests, most which require spinning up a new network instance, and expensive epoch transitions.
While I'm complaining about CI:
- It is the Year of Our Lord 2021. It is idiotic that the changelog fragment format is so stringent. Builds failing because of a trailing newline should not be a thing. And expecting developers to install the burning trashfire that is node.js and npm to avoid this is unrealistic.
- I for one try (or tried) to put effort into my github commit messages, why are changelog fragments a thing? A non-trivial part of my changelog fragment misery is because I'm copy/pasting from
git loginto a text file by hand (Read this as: Why isn't this automated?). - Having to create a PR to get a number to placate the changelog fragment linter, which kicks off CI, only to push a fragment (for trivial PRs) is wasteful.
Remedies (in no particular order):
- PRs that trigger an automated CI cycle should exclude dependabot, and perhaps flat out be limited by user.
- dependabot PRs should be mergable "as is" once they have gone through CI, and not "after someone adds one of those stupid fragment things", which requires(?) another CI cycle. (#4091)
- The Rust runtime client(s) used in tests should be rewritten in Go, to reduce build time. (#4038)
- More parts of the buildkite CI pipeline should be moved over to github actions (#3517).
- Our tests need to be re-architected so that they run faster. The problem got notably worse when I unified timekeeping (so perhaps making mock epoch transitions faster somehow is sufficient), but repeatedly spinning up/tearing down the network is not great.
- Consider reducing the number of tests that require SGX, or moving some of the "expected to pass" SGX tests to only run on merge.
- All use of grpcio should be removed from the codebase, and future use of the crate should be banned.
- "F*** it, we'll throw money at the problem". Add more agents.
- ci-lint should probably be skipped for draft PRs for frustration reduction.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Buildkite CI pipeline, the test execution setup, and ci-lint behavior for draft PRs; the issue names no specific files or tests. Review the referenced issues #3517, #4038, and #4091 to separate completed work from the remaining items. Done would require a scoped, measurable reduction in CI waiting or execution time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go, rust
- Domain
- build-system, ci-cd, devops, testing-qa
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100