ClickHouse / ClickHouse/ClickHouse
DDLWorker fails task execution with on cluster (since 25.8)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
https://www.innervate.com/
### Describe what's wrong
```
clickhouse2 :) set distributed_ddl_task_timeout=10;
clickhouse2 :) create table test on cluster region_us ( A Int64) engine=Log;
┌─host────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ clickhouse2 │ 9000 │ 0 │ │ 1 │ 0 │
└─────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘
1 row in set. Elapsed: 10.566 sec.
Received exception from server (version 25.12.4):
Code: 159. DB::Exception: Received from localhost:9000.
DB::Exception: Distributed DDL task /clickhouse/task_queue/ddl/query-0000000000 is not finished
on 1 of 2 hosts (0 of them are currently executing the task, 0 are inactive).
They are going to execute the query in background. Was waiting for 10.586000005 seconds,
which is longer than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED)
```
### Does it reproduce on the most recent release?
Yes
```
image: clickhouse/clickhouse-server:head
26.2.1.40
Received exception from server (version 26.2.1):
Code: 159. DB::Exception: Received from localhost:9000. DB::Exception:
Distributed DDL task /clickhouse/task_queue/ddl/query-0000000000 is not finished on 1 of 2 hosts
(0 of them are currently executing the task, 0 are inactive).
They are going to execute the query in background. Was waiting for 10.588249296 seconds, which is longer
than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED)
```
### How to reproduce
I use Distributed over Distributed tables to gather data from multiple independent clusters.
I have independent clusters EU, US, ASIA.
Each cluster have multiple shards. Multiple ClickHouse nodes.
Also each cluster has an additional record in the remote servers regions_all which contains all clusters (EU, US, ASIA) as shards.
The problem is that all nodes of a region are members of an additional dns record `region-us` which plays a role as a hostname in remote_servers to implement regions_all -> region-us+region-eu+region-asia (it works like this since Clickhouse version 20.3.).
This "common-region" dns record became an issue for `on cluster` since 25.8.
This cluster `regions_all` is only used for Distributed queries (to get data from all regions), it's never used for `on cluster`. But the host of this `regions_all` interfere with other `on cluster ...`.
```
$ nslookup region-us
....
Non-authoritative answer:
Name: region-us
Address: 172.23.0.12 ---<<<<<---- clickhouse2
Name: region-us
Address: 172.23.0.11 ---<<<<<---- clickhouse1
$ cat /etc/clickhouse-server/conf.d/cluster.xml
clickhouse1
clickhouse2
region-us ----<<<<<---- clickhouse1, clickhouse2, clickhouse....
select cluster, shard_num, host_name, is_local from system.clusters;
clickhouse1
┌─cluster─────┬─shard_num─┬─host_name───┬─is_local─┐
│ default │ 1 │ localhost │ 1 │
│ region_us │ 1 │ clickhouse1 │ 1 │
│ region_us │ 2 │ clickhouse2 │ 0 │
│ regions_all │ 1 │ region-us │ 1 │ ---<<-- all hosts of a region think they are host=region-us
└─────────────┴───────────┴─────────────┴──────────┘
clickhouse2
┌─cluster─────┬─shard_num─┬─host_name───┬─is_local─┐
│ default │ 1 │ localhost │ 1 │
│ region_us │ 1 │ clickhouse1 │ 0 │
│ region_us │ 2 │ clickhouse2 │ 1 │
│ regions_all │ 1 │ region-us │ 1 │ ---<<-- all hosts of a region think they are host=region-us
└─────────────┴───────────┴─────────────┴──────────┘
select * from system.zookeeper where path = '/clickhouse/task_queue/replicas';
┌─name─────────────┬─value─┬─path────────────────────────────┐
│ clickhouse2:9000 │ │ /clickhouse/task_queue/replicas │
│ region%2Dus:9000 │ │ /clickhouse/task_queue/replicas │ ----<<<<<----- culprit of the issue
│ clickhouse1:9000 │ │ /clickhouse/task_queue/replicas │
│ localhost:9000 │ │ /clickhouse/task_queue/replicas │
└──────────────────┴───────┴─────────────────────────────────┘
```
repro with docker-compose: [clickhouse-cluster-2-shards.tgz](https://github.com/user-attachments/files/24891225/clickhouse-cluster-2-shards.tgz)
------
before 25.8 everything works like a charm
```
image: clickhouse/clickhouse-server:25.7.8.71
```
```
clickhouse2 :) create table test on cluster region_us ( A Int64) engine=Log;
┌─host────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
1. │ clickhouse2 │ 9000 │ 0 │ │ 1 │ 0 │
2. │ clickhouse1 │ 9000 │ 0 │ │ 0 │ 0 │
└─────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘
2 rows in set. Elapsed: 0.078 sec.
```
https://github.com/ClickHouse/ClickHouse/issues/89693#issuecomment-3677936606
### Expected behavior
_No response_
### Error message and/or stacktrace
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.