ClickHouse / ClickHouse/ClickHouse

Stress tests fail to restart the server after `04323_text_index_marks_empty_part`: startup rejects a corrupted marks file that `ATTACH TABLE` tolerates

Open
#115,198 0 comments 0 reactions 1 assignee Claimed by @alexey-milovidov View on GitHub
comp-mergetree comp-text-index minor potential bug testing
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

Caused by: https://github.com/ClickHouse/ClickHouse/pull/114091
Related: https://github.com/ClickHouse/ClickHouse/pull/115036

### Describe what's wrong

Stress test jobs fail with `Cannot start clickhouse-server` when the run kills the server while `04323_text_index_marks_empty_part` is in its corruption window. The test deliberately appends one byte to `skp_idx_idx.mrk4` (24 → 25 bytes) and removes `checksums.txt` of the empty part, then re-attaches, queries and drops the table. If the server is killed between the corruption and the final `DROP`, the corrupted table stays on disk and the post-stress restart fails:

```
DB::Exception: Load job 'load table test_3.t_text_idx_empty' failed: Code: 246.
DB::Exception: Bad size of marks file '/var/lib/clickhouse/store/.../all_1_10_4/skp_idx_idx.mrk4': 25, must be: 24:
Cannot attach table `test_3`.`t_text_idx_empty` ... (CORRUPTED_DATA)
```

Four occurrences since 2026-08-14, all the same signature: PRs #113347, #100391, #76184 (`Stress test (amd_tsan)`/`(arm_tsan)`) and #115036 (`Stress test (arm_release)`, [report](https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=115036&sha=9156269951fd4455486a9ea61e5e9ac1e12e2ba1&name_0=PR&name_1=Stress%20test%20%28arm_release%29)). The window was widened by #114091 (merged 2026-08-10), which pinned the empty part with `remove_empty_parts = 0` — before that, the cleanup thread usually removed the corrupted empty part before a restart could trip over it, which is consistent with the failures starting on 2026-08-14.

Two ways to read it:

1. Server inconsistency: the running server *tolerates* this state — the test itself does `ATTACH TABLE`, `SYSTEM PREWARM MARK CACHE` and `SELECT` on the corrupted part successfully — but the startup metadata load rejects the very same on-disk state and fails the whole table load. Startup being stricter than `ATTACH` means any kill inside the window bricks the table until manual intervention.
2. Test hygiene: a `no-stress` tag is not supported by design ("stress tests must be able to run every test"), so the test must be kill-safe, and with `remove_empty_parts = 0` it is not — there is no point in its corruption window where a kill leaves a startup-loadable state.

### How to reproduce

* Run `04323_text_index_marks_empty_part`, kill the server after the `printf '\x00' >> .../skp_idx_idx.mrk4` / `rm checksums.txt` step (before the final `DROP TABLE`), and start the server again — table load fails with `CORRUPTED_DATA` while the same state was attachable before the kill.
* In CI: any stress job that kills the server inside that window.

No fiddle link: requires killing and restarting the server around on-disk state, which fiddle cannot do.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.