CI improvement for distributed & integration tests
- Dominant language
- C++
- Stars
- 3.6k
- Forks
- 468
- Avg merge
- 29m
- Merged PRs (30d)
- 1
Description
There are many limitations on our current CI:
- The environment on CI is hugely different from our laptop, it is hard to reproduce CI errors by our own;
- Especially for the integration tests on distributed setting, which involve k8s, (or Hadoop/Spark in the near future);
- Pain for debugging: a bugfix on distributed setting needs to package a new image, upload to registry, and then ask our "experts", i.e., Siyuan or Dongze for verification.
Hence the CI works like an obstacle for our development, however, it should be a productive tool.
Let's take a look at the [integration tests](https://github.com/ClickHouse/ClickHouse/tree/master/tests/integration) of Clickhouse, I think it was a good practice! It was super-easy to launch a test environment with a cluster with 4 nodes, each mount a mocked s3 disk, and a 3-nodes zookeeper, using a single-line command like
```bash
runner integration_test_s3
```

The Clickhouse integration CI workflow is shown in the fig-1:
1. developers build the binary on local machine,
2. by assigning `CLICKHOUSE_TESTS_SERVER_BIN_PATH` or arg to the binary built on step-1, the `runner` [code](https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/runner) will launch a docker container.
3. in the container, `pytest` will begin to run. Pay attention to its [helper](https://github.com/ClickHouse/ClickHouse/tree/master/tests/integration/helpers) functions. e.g., [cluster.py](https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/helpers/cluster.py) will launch a cluster of containers within this container, making it easier for distributed tests.
4. The logs for each instance will be exported to the local machine, for further debugging and investigation. shown in the fig-2.

GraphScope may borrow the ideas from this, to improve our CI.
Since Clickhouse builds the binary with static dependencies, its binary can be built on A and run on B. Whether we
1) need to link dependencies as static libraries;
2) or some adaptations need to be made (Prefered) :

How do you think? Any comments are welcome!
Strongly recommend to run a test following [README](https://github.com/ClickHouse/ClickHouse/tree/master/tests/integration), to see how complex the test setting is while keeps the user experience clean and easy.
Contributor guide
Assessment
This issue has not been assessed yet.