ClickHouse / ClickHouse/ClickHouse
Dropping a nested column group bypasses every ALTER protection
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Describe what's wrong
when shared nested offsets are enabled a name in `DROP COLUMN ` that is not a column of the table denotes the whole group of columns `.*` and the drop removes all of them
The stages of an `ALTER` disagree about this meaning: validation and the mutation stage treat the name as the group, while the metadata update and every protective check (dependent views, unfinished mutations, key columns, dependencies) compare column names exactly and do not see the group. Each mismatch below is one observable consequence
### Does it reproduce on the most recent release?
Yes
### How to reproduce
## 1. `DROP COLUMN IF EXISTS ` silently destroys the group's data
`ALTER` reports OK, `DESCRIBE` still shows `n.a` and `n.b` and the
`SELECT` returns `0 0 100`. The columns stay in the table while their values are gone
```sql
CREATE TABLE t (`n.a` UInt64, `n.b` UInt64, x UInt64) ENGINE = MergeTree ORDER BY x;
INSERT INTO t VALUES (1, 10, 100);
ALTER TABLE t DROP COLUMN IF EXISTS n;
SELECT * FROM t; -- 0 0 100
```
Fiddle: https://fiddle.clickhouse.com/7de96186-e49d-45c1-b8af-a56803993cd0
The same split also swallows pending updates:
```sql
CREATE TABLE t (`n.a` UInt64, x UInt64) ENGINE = MergeTree ORDER BY x;
INSERT INTO t VALUES (1, 1);
ALTER TABLE t UPDATE `n.a` = 2 WHERE 1 SETTINGS mutations_sync = 0;
ALTER TABLE t DROP COLUMN IF EXISTS n;
SELECT * FROM t; -- 0 1
```
## 2. A dependent materialized view does not protect the group's columns
```sql
CREATE TABLE src (`n.a` UInt64, `n.b` UInt64, x UInt64) ENGINE = MergeTree ORDER BY x;
CREATE MATERIALIZED VIEW mv ENGINE = Null AS SELECT `n.a` FROM src;
ALTER TABLE src DROP COLUMN n;
INSERT INTO src (x) VALUES (1);
```
Drop succeeds, and from that point every insert into `src` fails with `UNKNOWN_IDENTIFIER`, because the view still selects `n.a`. With `IF EXISTS` the outcome is the data destruction from problem 1, with the view still subscribed
Fiddle: https://fiddle.clickhouse.com/4dd388c7-af62-41d4-b097-bf4007e25041
## 3. An unfinished mutation on a group's column does not block dropping the group
```sql
CREATE TABLE t (`n.a` UInt64, x UInt64, c UInt64) ENGINE = MergeTree ORDER BY x;
INSERT INTO t VALUES (1, 1, 1);
ALTER TABLE t UPDATE c = `n.a` + 1 WHERE 1 SETTINGS mutations_sync = 0;
ALTER TABLE t DROP COLUMN n;
SELECT command, is_done, latest_fail_reason FROM system.mutations WHERE table = 't';
```
When the background mutation has not finished yet, the drop passes, the group is removed, and the queued mutation now references a column that no longer exists - it stays in the queue forever until `KILL MUTATION`. Whether the statement is safe depends on a race with the background pool
## 4. A column that the engine itself requires is not protected either
`CollapsingMergeTree` cannot work without its `sign` column, so dropping or clearing that column by its own name is forbidden. When the column belongs to a group, the group name is not recognised as that column and the command goes through, leaving the engine pointing at a column that no longer exists
```sql
CREATE TABLE t (`n.s` Int8, x UInt64) ENGINE = CollapsingMergeTree(`n.s`) ORDER BY x;
INSERT INTO t VALUES (1, 100);
ALTER TABLE t CLEAR COLUMN `n.s`; -- Code: 524. Trying to ALTER DROP sign (`n.s`) column. (ALTER_OF_COLUMN_IS_FORBIDDEN)
ALTER TABLE t CLEAR COLUMN n; -- OK, the sign column is zeroed
ALTER TABLE t DROP COLUMN n; -- OK, the sign column is gone
SHOW CREATE TABLE t; -- ENGINE = CollapsingMergeTree(`n.s`)
INSERT INTO t VALUES (5); -- Code: 10. Not found column n.s in block. There are only columns: x. (NOT_FOUND_COLUMN_IN_BLOCK)
DETACH TABLE t; ATTACH TABLE t; -- Code: 16. Sign column n.s does not exist in table declaration. (NO_SUCH_COLUMN_IN_TABLE)
```
`SELECT` still returns rows, so the table looks alive until the next insert, merge or restart - and after a restart it does not come back at all. The same applies to the `version` and `is_deleted` columns of `ReplacingMergeTree` and `VersionedCollapsingMergeTree`
### Expected behavior
_No response_
### Error message and/or stacktrace
_No response_
### Related issues and pull requests
_No response_
### Additional context
#114468 #114163
Contributor guide
Research direction
Reproduce the grouped DROP and CLEAR cases in the issue, including dependent views, pending mutations, and engine-required columns. Trace the ALTER validation, mutation, metadata-update, and protective-check stages to make them use the same nested-group meaning; done means destructive operations are blocked or handled consistently and the supplied scenarios no longer leave invalid metadata or queued work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100