alibaba / alibaba/GraphScope

Refactor and reorganize the full CI process

Open
#3,717 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
3.6k
Forks
468
Avg merge
29m
Merged PRs (30d)
1

Description

Currently, as you can see from the [workflow directory](https://github.com/alibaba/GraphScope/tree/main/.github/workflows), we have considerable CI processes, including:

- Minimal unit tests for each independent engine and storage
- Full nightly unit tests that run every night to reduce CI runtime for each PR
- E2E test for GraphScope on both local and K8s environment
- Building basic and development images automatically
- Cutting a release

However, with the influence in the community increases, we are facing some challenges, like

- PRs always keep failing inexplicably, even though it has nothing to do with the code they modified.
- Some nightly unit tests are in disrepair;
- Unit test for engines(especially GLE on macOS) is not enough. As a result, some failures will only appear during cutting a new release.

In additions, we are going to release GraphScope FLEX, which more CI tests will be brought in. And [Github runner](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners) already support macOS with M1 chip. So it's time for us to reorganize the CI process.

## Tasks:

The principle of this refactor of CI is to cover more tests on the basis of shortening the testing time of each PR as much as possible. So, we can simplify the pre-release testing process further. Tasks include

- Enrich unit test for each engine, including wheel package test, docker container test whether it is linux or macOS(both Intel and M1 chip) platform.
- Remove nightly test and split them into unit tests for each engine
- Support publishing GraphScope on Apple M1 chip automatically

After completing this task, you can expect the following benefits contributing to your personal growth:

- Systematically understand the construction and operation process of the GraphScope one-stop graph computing system.
- More comprehensive understand of different software building process, like cmake(cpp), maven(java), rustc(rust), and pip{python} on both linux and macOS(both Intel and M1 chip) platforms.
- You will explore new tools, technologies, and best practices in the industry field of CI/CD.
- Enhance your problem-solving, collaboration and communication skills as well as teamwork abilities.

Contributor guide

Open the contributing guide

Research direction

The issue points to the .github/workflows directory; start by inventorying the existing unit-test, nightly, E2E, image-building, and release workflows. Done means engine tests cover wheel and container packages on Linux and Intel/M1 macOS, nightly tests are removed or split into engine tests, PR testing is shorter, and Apple M1 publishing is automated.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, docker, github-actions, java, kubernetes, macos, python, rust
Domain
build-system, ci-cd, devops, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.