ClickHouse / ClickHouse/ClickHouse
CREATE TABLE w/ Replicated engine should fail when ZK node removal is scheduled
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
**Describe the issue**
When you DROP & CREATE **same** replicated tables in short time period, the newly CREATE'd table will be unusable after 8 minutes.
The reason is known. ZK node removal is asynchronous, and the zk node is removed after successful CREATE. The delay is configurable by `database_atomic_delay_before_drop_table_sec` settings. See discussion #23035
**How to reproduce**
* Which ClickHouse server version to use
21.3.4.25
* `CREATE TABLE` statements for all tables involved
```
CREATE TABLE default.tbl (
t DateTime
name String
)
ENGINE=ReplicatedMergeTree('/clickhouse/tables/{layer}-{shard}/default.tbl', '{replica}')
PARTITION BY toYYYYMMDD(t)
ORDER BY (t, name)
```
* Sample data for all these tables, use [clickhouse-obfuscator]
No data is necessary.
* Queries to run that lead to unexpected result
```
DROP TABLE tbl ON CLUSTER cluster_name
```
and run the create table again within 8min (default), check error log after 8min.
**Expected behavior**
In my opinion CREATE should fail when the same zookeeper node is scheduled for removal.
Or CREATE could cancel zk node removal if the zk node value is exactly same (primary key, order by, etc..)
Also it'd be nice to have scheduled removal time in one of the system tables for visibility.
**Error message and/or stacktrace**
```
2021.08.08 18:31:43.199248 [ 15947 ] {} default.tbl (6772ad45-d3c6-4375-b78a-0166d390f323): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999, e.displayText() = Coordination::Exception: Can't get data for node /clickhouse/tables/1-005/default.tbl/replicas/1r005/log_pointer: node doesn't exist (No node), Stack trace (when copying this message, always include the lines below):
0. Coordination::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, Coordination::Error, int) @ 0xfbfabe3 in /usr/bin/clickhouse
1. Coordination::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, Coordination::Error) @ 0xfbfae62 in /usr/bin/clickhouse
2. zkutil::ZooKeeper::get(std::__1::basic_string, std::__1::allocator > const&, Coordination::Stat*, std::__1::shared_ptr const&) @ 0xfc020f2 in /usr/bin/clickhouse
3. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::__1::shared_ptr, std::__1::function) @ 0xf7771a7 in /usr/bin/clickhouse
4. DB::StorageReplicatedMergeTree::queueUpdatingTask() @ 0xf40c11a in /usr/bin/clickhouse
5. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0xeadfe80 in /usr/bin/clickhouse
6. DB::BackgroundSchedulePool::threadFunction() @ 0xeae1e77 in /usr/bin/clickhouse
7. ? @ 0xeae2c42 in /usr/bin/clickhouse
8. ThreadPoolImpl::worker(std::__1::__list_iterator) @ 0x865165f in /usr/bin/clickhouse
9. ? @ 0x86550f3 in /usr/bin/clickhouse
10. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
11. clone @ 0x12171f in /lib/x86_64-linux-gnu/libc-2.27.so
(version 21.3.4.25 (official build))
```
**Additional context**
`SYNC` modifier for the DROP statement can avoid this problem but that's a side topic.
Contributor guide
Assessment
This issue has not been assessed yet.