ClickHouse / ClickHouse/ClickHouse
`ALTER RENAME COLUMN` racing with a merge still loses data on `26.3`: `04238_alter_rename_column_merge_race` fails ~12% there and 0% everywhere else
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
## Describe what's wrong
`ALTER TABLE ... RENAME COLUMN` racing with a concurrent merge can still lose the renamed
column's data on the `26.3` release branch. The regression test that covers exactly this,
`04238_alter_rename_column_merge_race`, fails there at a steady ~12% while it is green
everywhere else, including `master` and the `25.8` LTS.
The test does 25 iterations of: create a `MergeTree` table, insert 10 rows across 5 parts,
then run `ALTER TABLE t RENAME COLUMN d TO d1 SETTINGS alter_sync = 2` and
`OPTIMIZE TABLE t FINAL` concurrently, and check that all 10 rows still have a non-empty
`d1`. On `26.3` it fails with
```
FAIL on iteration N: expected 10 non-empty rows, got 0
```
Both statements return success; the renamed column simply comes back filled with defaults,
so this is silent data loss rather than an error.
## Evidence
Last 60 days, from `system.checks` on https://play.clickhouse.com (branch pushes only,
`pull_request_number = 0`):
| branch | job | OK | FAIL |
|---|---|---:|---:|
| `26.3` | `Stateless tests (amd_asan, distributed plan, parallel, 1/2)` | 196 | **27** |
| `25.8` | `Stateless tests (amd_asan, distributed plan, parallel, 1/2)` | 81 | 0 |
| `26.4` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel, 1/2)` | 51 | 0 |
| `26.5` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel, 1/2)` | 170 | 0 |
| `26.6` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel, 1/2)` | 307 | 0 |
| `26.7` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel)` | 259 | 0 |
| `26.8` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel)` | 138 | 0 |
| `master` | all configurations | 14510 | 0 |
Failures on `26.3` are spread over the whole window (2026-07-13 through 2026-09-05, 1-5 per
day), so this is not a one-off infrastructure event. One report:
https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=26.3&sha=f30ecf68c396adc27fc53467eedbddeb908b8109&name_0=ReleaseBranchCI&name_1=Stateless%20tests%20%28amd_asan%2C%20distributed%20plan%2C%20parallel%2C%201%2F2%29
The test file is byte-identical on `26.3` and `master`, so this is not a test-side
difference.
## Hypothesis
The first fix for this race, holding `currently_processing_in_background_mutex` across
`setProperties` and `startMutation` in `StorageMergeTree::alter` (issue #80648), **is**
present on `26.3` — `background_lock` is there.
What `26.3` does not have is the second fix, `Record a rename materialized by a merge in the
merged part's data version` (pull request #115981), which is on `master` only: its test
`05037_rename_column_materialized_by_merge.sql` is absent from both `26.3` and `25.8`.
That also explains why only `26.3` is red. The plain-`amd_asan` build is used for this
configuration on `25.8` and `26.3` only — every other branch and `master` run
`amd_asan_ubsan` there, and the extra UBSan instrumentation changes the timing enough to
hide the window. `25.8` runs the same plain-ASan build and is green because it predates the
second race. `26.3` is the one branch that both runs the revealing build and carries the
unfixed code.
If that is right, the fix is to backport #115981 to `26.3` (and to check whether
`26.4`-`26.8` carry the same unfixed code, merely hidden by their build flavour).
## How to reproduce
```
tests/queries/0_stateless/04238_alter_rename_column_merge_race.sh
```
against a `26.3` ASan build; it reproduces in roughly one run in eight.
Related: https://github.com/ClickHouse/ClickHouse/issues/80648
Related: https://github.com/ClickHouse/ClickHouse/pull/115981
Related: https://github.com/ClickHouse/ClickHouse/pull/104822
Contributor guide
Research direction
Start with tests/queries/0_stateless/04238_alter_rename_column_merge_race.sh and reproduce it on a 26.3 ASan build. Compare the 26.3 implementation with pull request #115981, including StorageMergeTree::alter and the merge data-version handling; done means the race test remains green across repeated runs without losing d1 values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100