Refactor and reorganize the full CI process
- Dominant language
- C++
- Stars
- 3.6k
- Forks
- 468
- Avg merge
- 29m
- Merged PRs (30d)
- 1
Description
Currently, as you can see from the [workflow directory](https://github.com/alibaba/GraphScope/tree/main/.github/workflows), we have considerable CI processes, including:
- Minimal unit tests for each independent engine and storage
- Full nightly unit tests that run every night to reduce CI runtime for each PR
- E2E test for GraphScope on both local and K8s environment
- Building basic and development images automatically
- Cutting a release
However, with the influence in the community increases, we are facing some challenges, like
- PRs always keep failing inexplicably, even though it has nothing to do with the code they modified.
- Some nightly unit tests are in disrepair;
- Unit test for engines(especially GLE on macOS) is not enough. As a result, some failures will only appear during cutting a new release.
In additions, we are going to release GraphScope FLEX, which more CI tests will be brought in. And [Github runner](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners) already support macOS with M1 chip. So it's time for us to reorganize the CI process.
## Tasks:
The principle of this refactor of CI is to cover more tests on the basis of shortening the testing time of each PR as much as possible. So, we can simplify the pre-release testing process further. Tasks include
- Enrich unit test for each engine, including wheel package test, docker container test whether it is linux or macOS(both Intel and M1 chip) platform.
- Remove nightly test and split them into unit tests for each engine
- Support publishing GraphScope on Apple M1 chip automatically
After completing this task, you can expect the following benefits contributing to your personal growth:
- Systematically understand the construction and operation process of the GraphScope one-stop graph computing system.
- More comprehensive understand of different software building process, like cmake(cpp), maven(java), rustc(rust), and pip{python} on both linux and macOS(both Intel and M1 chip) platforms.
- You will explore new tools, technologies, and best practices in the industry field of CI/CD.
- Enhance your problem-solving, collaboration and communication skills as well as teamwork abilities.
Contributor guide
Research direction
The issue points to the .github/workflows directory; start by inventorying the existing unit-test, nightly, E2E, image-building, and release workflows. Done means engine tests cover wheel and container packages on Linux and Intel/M1 macOS, nightly tests are removed or split into engine tests, PR testing is shorter, and Apple M1 publishing is automated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, docker, github-actions, java, kubernetes, macos, python, rust
- Domain
- build-system, ci-cd, devops, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100