ClickHouse / ClickHouse/dbt-clickhouse
Improve test performance
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
Issue created to track changes we made to increase test times after the findings on https://github.com/ClickHouse/dbt-clickhouse/issues/499
Currently we have two types of test depending on where the CH cluster is located:
- `test_matrix.yml`: Uses a local CH cluster inside a docker-compose cluster. Currently we execute in parallel a matrix of tests with dimensions: CH version and Python version. One extra dimension is done serially in each execution: connection being done with HTTP or Native protocols. Total time per job is around 10 minutes. this is not an issue right now but any improvement would be welcome
- `test_cloud.yml`: Uses a CH cluster hosted in CH cloud. Currently these tests are not being used as we have found some problems that made these tests took too much time (>2h with a lot of flakyness):
- The CH Cloud cluster was already overloaded.
- The DBT test cannot be currently run in parallel against the same CH cloud instance as some test were colliding between them.
Pending improvements:
- Use a new CH Cloud cluster (we are working on it right now)
- Test a way to be able to execute the CH Cloud test in different clusters (also being worked)
- Be able to execute test in parallel: in progress, it will take time.
Contributor guide
Research direction
Start by reviewing test_matrix.yml and test_cloud.yml, along with the findings in issue 499, to understand the current matrix dimensions, cluster setup, and test collisions. Measure the existing test behavior before changing it; done means the cloud tests can use the new cluster setup reliably and run in parallel without collisions or excessive runtime.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- ci-cd, databases, performance, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100