ClickHouse / ClickHouse/ClickHouse

Initiator query remains running and is unkillable when using icebergS3Cluster and underlying tasks have timeout

Open
#98,165 1 comment 0 reactions 1 assignee Claimed by @SmitaRKulkarni View on GitHub
comp-datalake external potential bug
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

_No response_

### Describe what's wrong

When loading big enough set of data from Iceberg into Clickhouse table using:
```
insert into select from icebergS3Cluster('{cluster}', ...)
```

It is possible for all underlying tasks to fail on

```
Code: 209. DB::Exception: Timeout exceeded while reading from socket (peer: [xxx]:52388, local: [xxx]:9000, 300000 ms): While executing ReadFromObjectStorage. (SOCKET_TIMEOUT) (version 25.8.2.29 (official build))
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000010c97930
1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000b735698
2. DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, long&&) @ 0x0000000010e7a82c
3. DB::ReadBufferFromPocoSocketBase::socketReceiveBytesImpl(char*, unsigned long) @ 0x0000000010e7a414
4. DB::ReadBufferFromPocoSocketBase::nextImpl() @ 0x0000000010e7aaec
5. DB::ReadBuffer::next() @ 0x0000000010d5ff70
6. std::shared_ptr std::__function::__policy_invoker ()>::__call_impl[abi:ne190107] ()>>(std::__function::__policy_storage const*) @ 0x0000000016639540
7. DB::StorageObjectStorageSource::ReadTaskIterator::next(unsigned long) @ 0x000000001375fd68
8. DB::StorageObjectStorageSource::createReader(unsigned long, std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr const&, DB::ReadFromFormatInfo&, std::optional const&, std::shared_ptr const&, DB::SchemaCache*, std::shared_ptr const&, unsigned long, std::shared_ptr, std::shared_ptr, bool) @ 0x0000000013757d68
9. DB::StorageObjectStorageSource::ReaderHolder std::__function::__policy_invoker::__call_impl[abi:ne190107]>(std::__function::__policy_storage const*) @ 0x0000000013769280
10. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, DB::StorageObjectStorageSource::ReaderHolder ()>::operator()() @ 0x00000000137665e0
11. void std::__function::__policy_invoker::__call_impl[abi:ne190107] (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda0'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000137668c4
12. ThreadPoolImpl>::ThreadFromThreadPool::worker() @ 0x0000000010dc7fbc
13. void std::__function::__policy_invoker::__call_impl[abi:ne190107]::ThreadFromGlobalPoolImpl>::ThreadFromThreadPool::*)(), ThreadPoolImpl>::ThreadFromThreadPool*>(void (ThreadPoolImpl>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000010dce3a0
14. ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x0000000010dc550c
15. void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x0000000010dcc51c
16. ? @ 0x000000000007d5b8
17. ? @ 0x00000000000e5edc
```

This leaves the initiator query still running in the cluster. `kill query` sets the flag for it to be `is_cancelled` but the process is still running. Only way to get rid of it is to restart the node.

I have no access to latest build where I can load that volume of the data.

### Does it reproduce on the most recent release?

No

### How to reproduce

Load sufficiently large data set with

```
insert into select from icebergS3Cluster('{cluster}', ...)
```

In our case its 170 files x 100MB of Parquet data.

### Expected behavior

_No response_

### Error message and/or stacktrace

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.