ClickHouse / ClickHouse/dbt-clickhouse
Fix the new set of Flaky tests randomly failing in CI
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
We are having several errors related to random tests failing. I have divided the failed tests in two sections:
### Tests that just need to wait until the data is replicated
Seems like in all these situations the problem is that the test is asserting that some data needs to be present but it request it to a replica that have not them yet. The main fix for them would be to retry the check until the data is replicated:
FAILED tests/integration/adapter/incremental/test_base_incremental.py::TestIncrementalCompoundKey::test_compound_key - assert 200 == 180
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/27113511086/job/80015787918
FAILED tests/integration/adapter/materialized_view/test_materialized_view.py::TestCatchup::test_full_refresh_catchup_enabled - assert 3 == 4
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26927605490/job/79440668311
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26861082807/job/79214569859
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26795644122/job/78991245933
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26732836076/job/78780023484
FAILED tests/integration/adapter/incremental/test_schema_change.py::TestOnSchemaChange::test_append[schema_change_append] - assert 4 == 5
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26615312343/job/78429598535
FAILED tests/integration/adapter/clickhouse/test_clickhouse_table_materializations.py::TestMergeTreeTableMaterialization - assert 10 >= 20
- https://github.com/ClickHouse/dbt-clickhouse/pull/639
### CH error codes that we need to retry
There are situations where one operation may fail because CH cannot process it in this particular moment. If the error is retryable, we should retry until the operation success.
Received ClickHouse exception, **code: 244**, server response: Code: 244. DB::Exception: Got unexpected ZooKeeper error ZNODEEXISTS (at index 10) for part all_0_0_0. (UNEXPECTED_ZOOKEEPER_ERROR) (for url https://***:8443)
- FAILED tests/integration/adapter/basic/test_ephemeral.py::TestEphemeral::test_ephemeral - AssertionError: dbt exit state did not match expected https://github.com/ClickHouse/dbt-clickhouse/actions/runs/27080994865/job/79926683096
- FAILED tests/integration/adapter/materialized_view/test_multiple_materialized_views_external_target.py::TestUpdateMultipleExternalTargetMVQueryOnly::test_update_mv_query - AssertionError: dbt exit state did not match expected https://github.com/ClickHouse/dbt-clickhouse/actions/runs/27032615573/job/80063403644
**Code: 242**. DB::Exception: Table is shutting down (zookeeper path: /clickhouse/tables/230e7b4e-a580-41ac-a1ee-2958c4a99a02/default). Stack trace
- FAILED tests/integration/adapter/materialized_view/test_refreshable_materialized_view_external_target.py::TestBasicExternalTargetRefreshableMV::test_create - AssertionError: dbt exit state did not match expected https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26701657425/job/78695437630
**Code: 279**, server response: Code: 279. DB::Exception: All connection tries failed. / **Code: 32**. DB::Exception: Attempt to read after eof. (ATTEMPT_TO_READ_AFTER_EOF
- FAILED tests/integration/adapter/clickhouse/test_adapter_response.py::TestAdapterResponseQueryId::test_query_id_round_trips_to_query_log - dbt_common.exceptions.base.DbtDatabaseError: Database Error https://github.com/ClickHouse/dbt-clickhouse/actions/runs/27032615573/job/79788485765
Contributor guide
Research direction
Start by running the listed failures in tests/integration/adapter/incremental/test_base_incremental.py, materialized_view/test_materialized_view.py, incremental/test_schema_change.py, clickhouse/test_clickhouse_table_materializations.py, and the other named test files. Trace how these tests check replicated data and handle ClickHouse errors 244, 242, 279, and 32. Done means the listed tests pass reliably, with retries for delayed replication and the specified retryable errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100