ClickHouse / ClickHouse/ClickHouse

`ALTER RENAME COLUMN` racing with a merge still loses data on `26.3`: `04238_alter_rename_column_merge_race` fails ~12% there and 0% everywhere else

Open
#119,382 1 comment 0 reactions 0 assignees View on GitHub
comp-mergetree
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

## Describe what's wrong

`ALTER TABLE ... RENAME COLUMN` racing with a concurrent merge can still lose the renamed
column's data on the `26.3` release branch. The regression test that covers exactly this,
`04238_alter_rename_column_merge_race`, fails there at a steady ~12% while it is green
everywhere else, including `master` and the `25.8` LTS.

The test does 25 iterations of: create a `MergeTree` table, insert 10 rows across 5 parts,
then run `ALTER TABLE t RENAME COLUMN d TO d1 SETTINGS alter_sync = 2` and
`OPTIMIZE TABLE t FINAL` concurrently, and check that all 10 rows still have a non-empty
`d1`. On `26.3` it fails with

```
FAIL on iteration N: expected 10 non-empty rows, got 0
```

Both statements return success; the renamed column simply comes back filled with defaults,
so this is silent data loss rather than an error.

## Evidence

Last 60 days, from `system.checks` on https://play.clickhouse.com (branch pushes only,
`pull_request_number = 0`):

| branch | job | OK | FAIL |
|---|---|---:|---:|
| `26.3` | `Stateless tests (amd_asan, distributed plan, parallel, 1/2)` | 196 | **27** |
| `25.8` | `Stateless tests (amd_asan, distributed plan, parallel, 1/2)` | 81 | 0 |
| `26.4` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel, 1/2)` | 51 | 0 |
| `26.5` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel, 1/2)` | 170 | 0 |
| `26.6` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel, 1/2)` | 307 | 0 |
| `26.7` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel)` | 259 | 0 |
| `26.8` | `Stateless tests (amd_asan_ubsan, distributed plan, parallel)` | 138 | 0 |
| `master` | all configurations | 14510 | 0 |

Failures on `26.3` are spread over the whole window (2026-07-13 through 2026-09-05, 1-5 per
day), so this is not a one-off infrastructure event. One report:
https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=26.3&sha=f30ecf68c396adc27fc53467eedbddeb908b8109&name_0=ReleaseBranchCI&name_1=Stateless%20tests%20%28amd_asan%2C%20distributed%20plan%2C%20parallel%2C%201%2F2%29

The test file is byte-identical on `26.3` and `master`, so this is not a test-side
difference.

## Hypothesis

The first fix for this race, holding `currently_processing_in_background_mutex` across
`setProperties` and `startMutation` in `StorageMergeTree::alter` (issue #80648), **is**
present on `26.3` — `background_lock` is there.

What `26.3` does not have is the second fix, `Record a rename materialized by a merge in the
merged part's data version` (pull request #115981), which is on `master` only: its test
`05037_rename_column_materialized_by_merge.sql` is absent from both `26.3` and `25.8`.

That also explains why only `26.3` is red. The plain-`amd_asan` build is used for this
configuration on `25.8` and `26.3` only — every other branch and `master` run
`amd_asan_ubsan` there, and the extra UBSan instrumentation changes the timing enough to
hide the window. `25.8` runs the same plain-ASan build and is green because it predates the
second race. `26.3` is the one branch that both runs the revealing build and carries the
unfixed code.

If that is right, the fix is to backport #115981 to `26.3` (and to check whether
`26.4`-`26.8` carry the same unfixed code, merely hidden by their build flavour).

## How to reproduce

```
tests/queries/0_stateless/04238_alter_rename_column_merge_race.sh
```
against a `26.3` ASan build; it reproduces in roughly one run in eight.

Related: https://github.com/ClickHouse/ClickHouse/issues/80648
Related: https://github.com/ClickHouse/ClickHouse/pull/115981
Related: https://github.com/ClickHouse/ClickHouse/pull/104822

Contributor guide

Open the contributing guide

Research direction

Start with tests/queries/0_stateless/04238_alter_rename_column_merge_race.sh and reproduce it on a 26.3 ASan build. Compare the 26.3 implementation with pull request #115981, including StorageMergeTree::alter and the merge data-version handling; done means the race test remains green across repeated runs without losing d1 values.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.