ClickHouse / ClickHouse/ClickHouse
Initiator query remains running and is unkillable when using icebergS3Cluster and underlying tasks have timeout
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Describe what's wrong
When loading big enough set of data from Iceberg into Clickhouse table using:
```
insert into select from icebergS3Cluster('{cluster}', ...)
```
It is possible for all underlying tasks to fail on
```
Code: 209. DB::Exception: Timeout exceeded while reading from socket (peer: [xxx]:52388, local: [xxx]:9000, 300000 ms): While executing ReadFromObjectStorage. (SOCKET_TIMEOUT) (version 25.8.2.29 (official build))
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000010c97930
1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000b735698
2. DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, long&&) @ 0x0000000010e7a82c
3. DB::ReadBufferFromPocoSocketBase::socketReceiveBytesImpl(char*, unsigned long) @ 0x0000000010e7a414
4. DB::ReadBufferFromPocoSocketBase::nextImpl() @ 0x0000000010e7aaec
5. DB::ReadBuffer::next() @ 0x0000000010d5ff70
6. std::shared_ptr std::__function::__policy_invoker ()>::__call_impl[abi:ne190107] ()>>(std::__function::__policy_storage const*) @ 0x0000000016639540
7. DB::StorageObjectStorageSource::ReadTaskIterator::next(unsigned long) @ 0x000000001375fd68
8. DB::StorageObjectStorageSource::createReader(unsigned long, std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr const&, DB::ReadFromFormatInfo&, std::optional const&, std::shared_ptr const&, DB::SchemaCache*, std::shared_ptr const&, unsigned long, std::shared_ptr, std::shared_ptr, bool) @ 0x0000000013757d68
9. DB::StorageObjectStorageSource::ReaderHolder std::__function::__policy_invoker::__call_impl[abi:ne190107]>(std::__function::__policy_storage const*) @ 0x0000000013769280
10. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, DB::StorageObjectStorageSource::ReaderHolder ()>::operator()() @ 0x00000000137665e0
11. void std::__function::__policy_invoker::__call_impl[abi:ne190107] (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda0'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000137668c4
12. ThreadPoolImpl>::ThreadFromThreadPool::worker() @ 0x0000000010dc7fbc
13. void std::__function::__policy_invoker::__call_impl[abi:ne190107]::ThreadFromGlobalPoolImpl>::ThreadFromThreadPool::*)(), ThreadPoolImpl>::ThreadFromThreadPool*>(void (ThreadPoolImpl>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000010dce3a0
14. ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x0000000010dc550c
15. void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x0000000010dcc51c
16. ? @ 0x000000000007d5b8
17. ? @ 0x00000000000e5edc
```
This leaves the initiator query still running in the cluster. `kill query` sets the flag for it to be `is_cancelled` but the process is still running. Only way to get rid of it is to restart the node.
I have no access to latest build where I can load that volume of the data.
### Does it reproduce on the most recent release?
No
### How to reproduce
Load sufficiently large data set with
```
insert into select from icebergS3Cluster('{cluster}', ...)
```
In our case its 170 files x 100MB of Parquet data.
### Expected behavior
_No response_
### Error message and/or stacktrace
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.