ClickHouse / ClickHouse/copier
Crashes during copying with clickhouse-copier
- Dominant language
- C++
- Stars
- 15
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Version 23.8.14 - latest in this LTS
Copying table from one cluster to another with clickhouse-copier. During last partion nodes in cluster are getting crashed
err.log
```
2024.06.06 11:15:22.017752 [ 31837 ] {} BaseDaemon: ########## Short fault info ############
2024.06.06 11:15:22.017839 [ 31837 ] {} BaseDaemon: (version 23.8.14.6 (official build), build id: 55188922FF1A7F7EDE14B7380AABEC65C2AF70A8, git hash: 967e51c1d6b9478a119a492ffa25449162854be2) (from thread 31329) Received signal 11
2024.06.06 11:15:22.017867 [ 31837 ] {} BaseDaemon: Signal description: Segmentation fault
2024.06.06 11:15:22.017895 [ 31837 ] {} BaseDaemon: Address: 0x306a. Access: read. Address not mapped to object.
2024.06.06 11:15:22.017930 [ 31837 ] {} BaseDaemon: Stack trace: 0x000000000712fa17 0x000000001241f74a 0x0000000012e49267 0x00000000134671b0 0x00000000133fe96b 0x00000000133fe69c 0x00000000133fdd93 0x0000000013467d3e 0x00000000131a147a 0x0000000013198270 0x00000
00013197a28 0x00000000131a96ca 0x000000001313bf4c 0x000000001314d759 0x0000000015b46614 0x0000000015b47811 0x0000000015c7e447 0x0000000015c7c71c 0x00007fcb780acea5 0x00007fcb77dd5b0d
2024.06.06 11:15:22.017955 [ 31837 ] {} BaseDaemon: ########################################
2024.06.06 11:15:22.017978 [ 31837 ] {} BaseDaemon: (version 23.8.14.6 (official build), build id: 55188922FF1A7F7EDE14B7380AABEC65C2AF70A8, git hash: 967e51c1d6b9478a119a492ffa25449162854be2) (from thread 31329) (query_id: e5ee8e01-4159-4090-8c28-7f0b8fe90ea7)
(query: INSERT INTO db.table_piece_0 (date, timestamp, aID, bID, cID, did, eID, colA, `nested.col1`, `nested.col2`, `nested.col3`, `nested.col4`, `nested.col5`, `nested.col6`, `nested.col7`, `nested.col8`, colB, colC, colD, `nested.col9`, `nested.col10`, colE) VALUES) Received signal Segmentation fault (11)
2024.06.06 11:15:22.018008 [ 31837 ] {} BaseDaemon: Address: 0x306a. Access: read. Address not mapped to object.
2024.06.06 11:15:22.018023 [ 31837 ] {} BaseDaemon: Stack trace: 0x000000000712fa17 0x000000001241f74a 0x0000000012e49267 0x00000000134671b0 0x00000000133fe96b 0x00000000133fe69c 0x00000000133fdd93 0x0000000013467d3e 0x00000000131a147a 0x0000000013198270 0x00000
00013197a28 0x00000000131a96ca 0x000000001313bf4c 0x000000001314d759 0x0000000015b46614 0x0000000015b47811 0x0000000015c7e447 0x0000000015c7c71c 0x00007fcb780acea5 0x00007fcb77dd5b0d
2024.06.06 11:15:22.018720 [ 31837 ] {} BaseDaemon: 2. memcpy @ 0x000000000712fa17 in /usr/bin/clickhouse
2024.06.06 11:15:22.019115 [ 31837 ] {} BaseDaemon: 3. DB::ColumnString::insertRangeFrom(DB::IColumn const&, unsigned long, unsigned long) @ 0x000000001241f74a in /usr/bin/clickhouse
2024.06.06 11:15:22.019311 [ 31837 ] {} BaseDaemon: 4. DB::Block DB::SquashingTransform::addImpl(DB::Block&&) @ 0x0000000012e49267 in /usr/bin/clickhouse
2024.06.06 11:15:22.019480 [ 31837 ] {} BaseDaemon: 5. DB::SquashingChunksTransform::onConsume(DB::Chunk) @ 0x00000000134671b0 in /usr/bin/clickhouse
2024.06.06 11:15:22.019997 [ 31837 ] {} BaseDaemon: 6. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x00000000
133fe96b in /usr/bin/clickhouse
2024.06.06 11:15:22.020048 [ 31837 ] {} BaseDaemon: 7. DB::runStep(std::function, DB::ThreadStatus*, std::atomic*) @ 0x00000000133fe69c in /usr/bin/clickhouse
2024.06.06 11:15:22.020077 [ 31837 ] {} BaseDaemon: 8. DB::ExceptionKeepingTransform::work() @ 0x00000000133fdd93 in /usr/bin/clickhouse
2024.06.06 11:15:22.020114 [ 31837 ] {} BaseDaemon: 9. DB::SquashingChunksTransform::work() @ 0x0000000013467d3e in /usr/bin/clickhouse
2024.06.06 11:15:22.020239 [ 31837 ] {} BaseDaemon: 10. DB::ExecutionThreadContext::executeTask() @ 0x00000000131a147a in /usr/bin/clickhouse
2024.06.06 11:15:22.020273 [ 31837 ] {} BaseDaemon: 11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x0000000013198270 in /usr/bin/clickhouse
2024.06.06 11:15:22.020298 [ 31837 ] {} BaseDaemon: 12. DB::PipelineExecutor::executeStep(std::atomic*) @ 0x0000000013197a28 in /usr/bin/clickhouse
2024.06.06 11:15:22.020376 [ 31837 ] {} BaseDaemon: 13. DB::PushingPipelineExecutor::push(DB::Block) @ 0x00000000131a96ca in /usr/bin/clickhouse
2024.06.06 11:15:22.020501 [ 31837 ] {} BaseDaemon: 14. DB::TCPHandler::runImpl() @ 0x000000001313bf4c in /usr/bin/clickhouse
2024.06.06 11:15:22.020525 [ 31837 ] {} BaseDaemon: 15. DB::TCPHandler::run() @ 0x000000001314d759 in /usr/bin/clickhouse
2024.06.06 11:15:22.020721 [ 31837 ] {} BaseDaemon: 16. Poco::Net::TCPServerConnection::start() @ 0x0000000015b46614 in /usr/bin/clickhouse
2024.06.06 11:15:22.020739 [ 31837 ] {} BaseDaemon: 17. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b47811 in /usr/bin/clickhouse
2024.06.06 11:15:22.020859 [ 31837 ] {} BaseDaemon: 18. Poco::PooledThread::run() @ 0x0000000015c7e447 in /usr/bin/clickhouse
2024.06.06 11:15:22.020880 [ 31837 ] {} BaseDaemon: 19. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7c71c in /usr/bin/clickhouse
2024.06.06 11:15:22.021147 [ 31837 ] {} BaseDaemon: 20. start_thread @ 0x0000000000007ea5 in /usr/lib64/libpthread-2.17.so
2024.06.06 11:15:22.021291 [ 31837 ] {} BaseDaemon: 21. clone @ 0x00000000000feb0d in /usr/lib64/libc-2.17.so
2024.06.06 11:15:22.399425 [ 31837 ] {} BaseDaemon: Integrity check of the executable successfully passed (checksum: EC42AF1A914504FFE44C59D96B8633E0)
2024.06.06 11:15:22.400943 [ 31837 ] {} BaseDaemon: Report this error to https://github.com/ClickHouse/ClickHouse/issues
2024.06.06 11:15:22.401049 [ 31837 ] {} BaseDaemon: Changed settings: use_uncompressed_cache = false, alter_sync = 2, load_balancing = 'random', log_queries = true, insert_distributed_sync = true, joined_subquery_requires_alias = true, max_memory_usage = 30000000000, send_logs_level = 'none', prefer_localhost_replica = false
2024.06.06 11:15:26.612877 [ 28299 ] {} Application: Child process was terminated by signal 11.
```
Entries in system.crash_log
```
Row 1:
──────
event_date: 2024-06-06
event_time: 2024-06-06 11:15:22
timestamp_ns: 1717665322400055688
signal: 11
thread_id: 31329
query_id: e5ee8e01-4159-4090-8c28-7f0b8fe90ea7
trace: [118684183,306313034,316969575,323383728,322955627,322954908,322952595,323386686,320476282,320438896,320436776,320509642,320061260,320132953,364144148,364148753,365421639,365414172,140511869062821,140511866084109]
trace_full: ['2. memcpy @ 0x000000000712fa17 in /usr/bin/clickhouse','3. DB::ColumnString::insertRangeFrom(DB::IColumn const&, unsigned long, unsigned long) @ 0x000000001241f74a in /usr/bin/clickhouse','4. DB::Block DB::SquashingTransform::addImpl(DB::Block&&) @ 0x0000000012e49267 in /usr/bin/clickhouse','5. DB::SquashingChunksTransform::onConsume(DB::Chunk) @ 0x00000000134671b0 in /usr/bin/clickhouse','6. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x00000000133fe96b in /usr/bin/clickhouse','7. DB::runStep(std::function, DB::ThreadStatus*, std::atomic*) @ 0x00000000133fe69c in /usr/bin/clickhouse','8. DB::ExceptionKeepingTransform::work() @ 0x00000000133fdd93 in /usr/bin/clickhouse','9. DB::SquashingChunksTransform::work() @ 0x0000000013467d3e in /usr/bin/clickhouse','10. DB::ExecutionThreadContext::executeTask() @ 0x00000000131a147a in /usr/bin/clickhouse','11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x0000000013198270 in /usr/bin/clickhouse','12. DB::PipelineExecutor::executeStep(std::atomic*) @ 0x0000000013197a28 in /usr/bin/clickhouse','13. DB::PushingPipelineExecutor::push(DB::Block) @ 0x00000000131a96ca in /usr/bin/clickhouse','14. DB::TCPHandler::runImpl() @ 0x000000001313bf4c in /usr/bin/clickhouse','15. DB::TCPHandler::run() @ 0x000000001314d759 in /usr/bin/clickhouse','16. Poco::Net::TCPServerConnection::start() @ 0x0000000015b46614 in /usr/bin/clickhouse','17. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b47811 in /usr/bin/clickhouse','18. Poco::PooledThread::run() @ 0x0000000015c7e447 in /usr/bin/clickhouse','19. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7c71c in /usr/bin/clickhouse','20. start_thread @ 0x0000000000007ea5 in /usr/lib64/libpthread-2.17.so','21. clone @ 0x00000000000feb0d in /usr/lib64/libc-2.17.so']
version: ClickHouse 23.8.14.6
revision: 54477
build_id: 55188922FF1A7F7EDE14B7380AABEC65C2AF70A8
Row 2:
──────
event_date: 2024-06-06
event_time: 2024-06-06 10:57:28
timestamp_ns: 1717664248440411481
signal: 11
thread_id: 1576
query_id: dbd8be6e-8f3d-4ad2-8be6-56c5700fae2d
trace: [118628695,306238634,316880007,323306224,322878123,322877404,322875091,323309182,320398778,320361456,320359336,320432138,319983820,320055513,364065876,364070481,365342343,365334876,139887600889509,139887597910797]
trace_full: ['2. memcpy @ 0x0000000007122157 in /usr/bin/clickhouse','3. DB::ColumnString::insertRangeFrom(DB::IColumn const&, unsigned long, unsigned long) @ 0x000000001240d4aa in /usr/bin/clickhouse','4. DB::Block DB::SquashingTransform::addImpl(DB::Block&&) @ 0x0000000012e33487 in /usr/bin/clickhouse','5. DB::SquashingChunksTransform::onConsume(DB::Chunk) @ 0x00000000134542f0 in /usr/bin/clickhouse','6. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x00000000133ebaab in /usr/bin/clickhouse','7. DB::runStep(std::function, DB::ThreadStatus*, std::atomic*) @ 0x00000000133eb7dc in /usr/bin/clickhouse','8. DB::ExceptionKeepingTransform::work() @ 0x00000000133eaed3 in /usr/bin/clickhouse','9. DB::SquashingChunksTransform::work() @ 0x0000000013454e7e in /usr/bin/clickhouse','10. DB::ExecutionThreadContext::executeTask() @ 0x000000001318e5ba in /usr/bin/clickhouse','11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x00000000131853f0 in /usr/bin/clickhouse','12. DB::PipelineExecutor::executeStep(std::atomic*) @ 0x0000000013184ba8 in /usr/bin/clickhouse','13. DB::PushingPipelineExecutor::push(DB::Block) @ 0x000000001319680a in /usr/bin/clickhouse','14. DB::TCPHandler::runImpl() @ 0x00000000131290cc in /usr/bin/clickhouse','15. DB::TCPHandler::run() @ 0x000000001313a8d9 in /usr/bin/clickhouse','16. Poco::Net::TCPServerConnection::start() @ 0x0000000015b33454 in /usr/bin/clickhouse','17. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b34651 in /usr/bin/clickhouse','18. Poco::PooledThread::run() @ 0x0000000015c6ae87 in /usr/bin/clickhouse','19. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c6915c in /usr/bin/clickhouse','20. start_thread @ 0x0000000000007ea5 in /usr/lib64/libpthread-2.17.so','21. clone @ 0x00000000000feb0d in /usr/lib64/libc-2.17.so']
version: ClickHouse 23.8.9.54
revision: 54477
build_id: 0C3272D074DC3F5673C4A885B0F9B0B5A0DA3609
```
Copied table on destination host
```
CREATE TABLE db.table
(
`date` Date DEFAULT CAST('1970-01-01', 'Date'),
`timestamp` DateTime DEFAULT CAST(-3600, 'DateTime'),
`aID` String,
`bID` String,
`cID` String,
`did` String,
`eID` String,
`colA` String,
`nested.col1` Array(String),
`nested.col2` Array(String),
`nested.col3` Array(String),
`nested.col4` Array(String),
`nested.col5` Array(String),
`nested.col6` Array(Float64),
`nested.col7` Array(String),
`nested.col8` Array(String),
`colB` String,
`colC` String,
`colD` String,
`nested.col9` Array(Float64),
`nested.col10` Array(Float64),
`colE` LowCardinality(String)
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/db/table', '{replica}')
PARTITION BY toYYYYMM(date)
ORDER BY (date, colA, did)
SETTINGS index_granularity = 8192
```
Contributor guide
Assessment
This issue has not been assessed yet.