ClickHouse / ClickHouse/dbt-clickhouse

Fix the new set of Flaky tests randomly failing in CI

Open
#666 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

We are having several errors related to random tests failing. I have divided the failed tests in two sections:

### Tests that just need to wait until the data is replicated
Seems like in all these situations the problem is that the test is asserting that some data needs to be present but it request it to a replica that have not them yet. The main fix for them would be to retry the check until the data is replicated:

FAILED tests/integration/adapter/incremental/test_base_incremental.py::TestIncrementalCompoundKey::test_compound_key - assert 200 == 180
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/27113511086/job/80015787918

FAILED tests/integration/adapter/materialized_view/test_materialized_view.py::TestCatchup::test_full_refresh_catchup_enabled - assert 3 == 4
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26927605490/job/79440668311
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26861082807/job/79214569859
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26795644122/job/78991245933
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26732836076/job/78780023484

FAILED tests/integration/adapter/incremental/test_schema_change.py::TestOnSchemaChange::test_append[schema_change_append] - assert 4 == 5
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26615312343/job/78429598535

FAILED tests/integration/adapter/clickhouse/test_clickhouse_table_materializations.py::TestMergeTreeTableMaterialization - assert 10 >= 20
- https://github.com/ClickHouse/dbt-clickhouse/pull/639

### CH error codes that we need to retry
There are situations where one operation may fail because CH cannot process it in this particular moment. If the error is retryable, we should retry until the operation success.

Received ClickHouse exception, **code: 244**, server response: Code: 244. DB::Exception: Got unexpected ZooKeeper error ZNODEEXISTS (at index 10) for part all_0_0_0. (UNEXPECTED_ZOOKEEPER_ERROR) (for url https://***:8443)
- FAILED tests/integration/adapter/basic/test_ephemeral.py::TestEphemeral::test_ephemeral - AssertionError: dbt exit state did not match expected https://github.com/ClickHouse/dbt-clickhouse/actions/runs/27080994865/job/79926683096
- FAILED tests/integration/adapter/materialized_view/test_multiple_materialized_views_external_target.py::TestUpdateMultipleExternalTargetMVQueryOnly::test_update_mv_query - AssertionError: dbt exit state did not match expected https://github.com/ClickHouse/dbt-clickhouse/actions/runs/27032615573/job/80063403644

**Code: 242**. DB::Exception: Table is shutting down (zookeeper path: /clickhouse/tables/230e7b4e-a580-41ac-a1ee-2958c4a99a02/default). Stack trace
- FAILED tests/integration/adapter/materialized_view/test_refreshable_materialized_view_external_target.py::TestBasicExternalTargetRefreshableMV::test_create - AssertionError: dbt exit state did not match expected https://github.com/ClickHouse/dbt-clickhouse/actions/runs/26701657425/job/78695437630

**Code: 279**, server response: Code: 279. DB::Exception: All connection tries failed. / **Code: 32**. DB::Exception: Attempt to read after eof. (ATTEMPT_TO_READ_AFTER_EOF
- FAILED tests/integration/adapter/clickhouse/test_adapter_response.py::TestAdapterResponseQueryId::test_query_id_round_trips_to_query_log - dbt_common.exceptions.base.DbtDatabaseError: Database Error https://github.com/ClickHouse/dbt-clickhouse/actions/runs/27032615573/job/79788485765

Contributor guide

Open the contributing guide

Research direction

Start by running the listed failures in tests/integration/adapter/incremental/test_base_incremental.py, materialized_view/test_materialized_view.py, incremental/test_schema_change.py, clickhouse/test_clickhouse_table_materializations.py, and the other named test files. Trace how these tests check replicated data and handle ClickHouse errors 244, 242, 279, and 32. Done means the listed tests pass reliably, with retries for delayed replication and the specified retryable errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.