ClickHouse / ClickHouse/ClickHouse

DDLWorker fails task execution with on cluster (since 25.8)

Open
#95,316 7 comments 1 reaction 1 assignee Claimed by @tuanpach View on GitHub
comp-ddl external potential bug
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

https://www.innervate.com/

### Describe what's wrong

```
clickhouse2 :) set distributed_ddl_task_timeout=10;

clickhouse2 :) create table test on cluster region_us ( A Int64) engine=Log;

┌─host────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ clickhouse2 │ 9000 │ 0 │ │ 1 │ 0 │
└─────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

1 row in set. Elapsed: 10.566 sec.

Received exception from server (version 25.12.4):
Code: 159. DB::Exception: Received from localhost:9000.
DB::Exception: Distributed DDL task /clickhouse/task_queue/ddl/query-0000000000 is not finished
on 1 of 2 hosts (0 of them are currently executing the task, 0 are inactive).
They are going to execute the query in background. Was waiting for 10.586000005 seconds,
which is longer than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED)
```

### Does it reproduce on the most recent release?

Yes

```
image: clickhouse/clickhouse-server:head

26.2.1.40

Received exception from server (version 26.2.1):
Code: 159. DB::Exception: Received from localhost:9000. DB::Exception:
Distributed DDL task /clickhouse/task_queue/ddl/query-0000000000 is not finished on 1 of 2 hosts
(0 of them are currently executing the task, 0 are inactive).
They are going to execute the query in background. Was waiting for 10.588249296 seconds, which is longer
than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED)
```

### How to reproduce

I use Distributed over Distributed tables to gather data from multiple independent clusters.

I have independent clusters EU, US, ASIA.
Each cluster have multiple shards. Multiple ClickHouse nodes.
Also each cluster has an additional record in the remote servers regions_all which contains all clusters (EU, US, ASIA) as shards.

The problem is that all nodes of a region are members of an additional dns record `region-us` which plays a role as a hostname in remote_servers to implement regions_all -> region-us+region-eu+region-asia (it works like this since Clickhouse version 20.3.).

This "common-region" dns record became an issue for `on cluster` since 25.8.

This cluster `regions_all` is only used for Distributed queries (to get data from all regions), it's never used for `on cluster`. But the host of this `regions_all` interfere with other `on cluster ...`.

```
$ nslookup region-us
....
Non-authoritative answer:
Name: region-us
Address: 172.23.0.12 ---<<<<<---- clickhouse2
Name: region-us
Address: 172.23.0.11 ---<<<<<---- clickhouse1

$ cat /etc/clickhouse-server/conf.d/cluster.xml




clickhouse1




clickhouse2





region-us ----<<<<<---- clickhouse1, clickhouse2, clickhouse....


select cluster, shard_num, host_name, is_local from system.clusters;

clickhouse1
┌─cluster─────┬─shard_num─┬─host_name───┬─is_local─┐
│ default │ 1 │ localhost │ 1 │
│ region_us │ 1 │ clickhouse1 │ 1 │
│ region_us │ 2 │ clickhouse2 │ 0 │
│ regions_all │ 1 │ region-us │ 1 │ ---<<-- all hosts of a region think they are host=region-us
└─────────────┴───────────┴─────────────┴──────────┘

clickhouse2
┌─cluster─────┬─shard_num─┬─host_name───┬─is_local─┐
│ default │ 1 │ localhost │ 1 │
│ region_us │ 1 │ clickhouse1 │ 0 │
│ region_us │ 2 │ clickhouse2 │ 1 │
│ regions_all │ 1 │ region-us │ 1 │ ---<<-- all hosts of a region think they are host=region-us
└─────────────┴───────────┴─────────────┴──────────┘

select * from system.zookeeper where path = '/clickhouse/task_queue/replicas';
┌─name─────────────┬─value─┬─path────────────────────────────┐
│ clickhouse2:9000 │ │ /clickhouse/task_queue/replicas │
│ region%2Dus:9000 │ │ /clickhouse/task_queue/replicas │ ----<<<<<----- culprit of the issue
│ clickhouse1:9000 │ │ /clickhouse/task_queue/replicas │
│ localhost:9000 │ │ /clickhouse/task_queue/replicas │
└──────────────────┴───────┴─────────────────────────────────┘
```

repro with docker-compose: [clickhouse-cluster-2-shards.tgz](https://github.com/user-attachments/files/24891225/clickhouse-cluster-2-shards.tgz)

------

before 25.8 everything works like a charm

```
image: clickhouse/clickhouse-server:25.7.8.71
```

```
clickhouse2 :) create table test on cluster region_us ( A Int64) engine=Log;

┌─host────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
1. │ clickhouse2 │ 9000 │ 0 │ │ 1 │ 0 │
2. │ clickhouse1 │ 9000 │ 0 │ │ 0 │ 0 │
└─────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

2 rows in set. Elapsed: 0.078 sec.
```

https://github.com/ClickHouse/ClickHouse/issues/89693#issuecomment-3677936606

### Expected behavior

_No response_

### Error message and/or stacktrace

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.