ClickHouse / ClickHouse/ClickHouse

When using distributed writing, data may be distributed to nodes that are offline, resulting in data retaining in the local filesystem.

Open
#65,954 1 comment 0 reactions 0 assignees View on GitHub
comp-distributed external operations question
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

When using distributed writing, data may be distributed to nodes that are offline, resulting in data being still in the local filesystem.
Encountering problematic versions: 21.8.3 22.3.2 23.8.3

relate: https://github.com/ClickHouse/ClickHouse/issues/60990

my old cluster config:
select cluster,shard_num,host_address from system.clusters;
```
┌─cluster───────────────────┬─shard_num─┬─host_address─┐
│ vip_order_analysis │ 1 │ 10.74.58.16 │
│ vip_order_analysis │ 1 │ 10.74.58.15 │
│ vip_order_analysis │ 2 │ 10.74.58.18 │
│ vip_order_analysis │ 2 │ 10.74.58.17 │
│ vip_order_analysis │ 3 │ 10.74.58.20 │
│ vip_order_analysis │ 3 │ 10.74.58.19 │
│ vip_order_analysis │ 4 │ 10.74.58.22 │
│ vip_order_analysis │ 4 │ 10.74.58.21 │
│ vip_order_analysis │ 5 │ 10.74.58.24 │
│ vip_order_analysis │ 5 │ 10.74.58.23 │
│ vip_order_analysis │ 6 │ 10.74.58.26 │
│ vip_order_analysis │ 6 │ 10.74.58.25 │
│ vip_order_analysis │ 7 │ 10.74.58.28 │
│ vip_order_analysis │ 7 │ 10.74.58.27 │
│ vip_order_analysis │ 8 │ 10.74.58.29 │
│ vip_order_analysis │ 8 │ 10.74.58.30 │
│ vip_order_analysis │ 9 │ 10.74.58.32 │
│ vip_order_analysis │ 9 │ 10.74.58.31 │
│ vip_order_analysis │ 10 │ 10.74.58.34 │
│ vip_order_analysis │ 10 │ 10.74.58.33 │
│ vip_order_analysis-onerep │ 1 │ 10.74.58.16 │
│ vip_order_analysis-onerep │ 2 │ 10.74.58.15 │
│ vip_order_analysis-onerep │ 3 │ 10.74.58.18 │
│ vip_order_analysis-onerep │ 4 │ 10.74.58.17 │
│ vip_order_analysis-onerep │ 5 │ 10.74.58.20 │
│ vip_order_analysis-onerep │ 6 │ 10.74.58.19 │
│ vip_order_analysis-onerep │ 7 │ 10.74.58.22 │
│ vip_order_analysis-onerep │ 8 │ 10.74.58.21 │
│ vip_order_analysis-onerep │ 9 │ 10.74.58.24 │
│ vip_order_analysis-onerep │ 10 │ 10.74.58.23 │
│ vip_order_analysis-onerep │ 11 │ 10.74.58.26 │
│ vip_order_analysis-onerep │ 12 │ 10.74.58.25 │
│ vip_order_analysis-onerep │ 13 │ 10.74.58.28 │
│ vip_order_analysis-onerep │ 14 │ 10.74.58.27 │
│ vip_order_analysis-onerep │ 15 │ 10.74.58.29 │
│ vip_order_analysis-onerep │ 16 │ 10.74.58.30 │
│ vip_order_analysis-onerep │ 17 │ 10.74.58.32 │
│ vip_order_analysis-onerep │ 18 │ 10.74.58.31 │
│ vip_order_analysis-onerep │ 19 │ 10.74.58.34 │
│ vip_order_analysis-onerep │ 20 │ 10.74.58.33 │
└───────────────────────────┴───────────┴──────────────┘
```
now cluster config :
select cluster,shard_num,host_address from system.clusters;

```
┌─cluster───────────────────┬─shard_num─┬─host_address─┐
│ vip_order_analysis │ 1 │ 10.74.58.16 │
│ vip_order_analysis │ 1 │ 10.74.58.15 │
│ vip_order_analysis │ 2 │ 10.74.58.18 │
│ vip_order_analysis │ 2 │ 10.74.58.17 │
│ vip_order_analysis │ 3 │ 10.74.58.20 │
│ vip_order_analysis │ 3 │ 10.74.58.19 │
│ vip_order_analysis │ 4 │ 10.74.58.22 │
│ vip_order_analysis │ 4 │ 10.74.58.21 │
│ vip_order_analysis │ 5 │ 10.74.58.24 │
│ vip_order_analysis │ 5 │ 10.74.58.23 │
│ vip_order_analysis │ 6 │ 10.74.58.26 │
│ vip_order_analysis │ 6 │ 10.74.58.25 │
│ vip_order_analysis │ 7 │ 10.74.58.28 │
│ vip_order_analysis │ 7 │ 10.74.58.27 │
│ vip_order_analysis │ 8 │ 10.74.58.29 │
│ vip_order_analysis │ 8 │ 10.74.49.1 │
│ vip_order_analysis │ 9 │ 10.74.58.32 │
│ vip_order_analysis │ 9 │ 10.74.58.31 │
│ vip_order_analysis │ 10 │ 10.74.58.34 │
│ vip_order_analysis │ 10 │ 10.74.58.33 │
│ vip_order_analysis-onerep │ 1 │ 10.74.58.16 │
│ vip_order_analysis-onerep │ 2 │ 10.74.58.15 │
│ vip_order_analysis-onerep │ 3 │ 10.74.58.18 │
│ vip_order_analysis-onerep │ 4 │ 10.74.58.17 │
│ vip_order_analysis-onerep │ 5 │ 10.74.58.20 │
│ vip_order_analysis-onerep │ 6 │ 10.74.58.19 │
│ vip_order_analysis-onerep │ 7 │ 10.74.58.22 │
│ vip_order_analysis-onerep │ 8 │ 10.74.58.21 │
│ vip_order_analysis-onerep │ 9 │ 10.74.58.24 │
│ vip_order_analysis-onerep │ 10 │ 10.74.58.23 │
│ vip_order_analysis-onerep │ 11 │ 10.74.58.26 │
│ vip_order_analysis-onerep │ 12 │ 10.74.58.25 │
│ vip_order_analysis-onerep │ 13 │ 10.74.58.28 │
│ vip_order_analysis-onerep │ 14 │ 10.74.58.27 │
│ vip_order_analysis-onerep │ 15 │ 10.74.58.29 │
│ vip_order_analysis-onerep │ 16 │ 10.74.49.1 │
│ vip_order_analysis-onerep │ 17 │ 10.74.58.32 │
│ vip_order_analysis-onerep │ 18 │ 10.74.58.31 │
│ vip_order_analysis-onerep │ 19 │ 10.74.58.34 │
│ vip_order_analysis-onerep │ 20 │ 10.74.58.33 │
└───────────────────────────┴───────────┴──────────────┘

```
parts metrika.xml
```




1
true

10.74.58.16
9087
xxxx
xxxx


10.74.58.15
9087
xxxx
xxxxxx


```

One day, node 10.74.58.30 suddenly broke down and was replaced by node 10.74.49.1. When writing data through the distributed table, there is a chance that the distributed table will not distribute the data to 10.74.49.1, but instead distribute it to the original node 10.74.58.30, which no longer exists in the cluster. This leads to some data not being written to the local table and results in data loss. This issue was present in versions 21.11.3 and 22.3.2, and even after upgrading to version 23.8.3-lts, the problem still persists.

I have checked all the metrika.xml files and they are already using the new address 10.74.49.1. The system.clusters file has also been updated with the new address 10.74.49.1. I can confirm that the configuration is correct without any issues.

about logs:
```
2024.07.02 10:05:03.819418 [ 33289 ] {2940e0c9-e4c3-4548-ab96-98931a0a60b3} executeQuery: Read 5001 rows, 371.17 KiB in 0.01535 sec., 325798.04560260585 rows/sec., 23.61 MiB/sec.
2024.07.02 10:05:03.819493 [ 33289 ] {2940e0c9-e4c3-4548-ab96-98931a0a60b3} DynamicQueryHandler: Done processing query
2024.07.02 10:05:03.819505 [ 33289 ] {2940e0c9-e4c3-4548-ab96-98931a0a60b3} MemoryTracker: Peak memory usage (for query): 4.78 MiB.
2024.07.02 10:05:03.819515 [ 33289 ] {} HTTP-Session: 1eb02403-9a43-429e-ab0e-db1ad4016971 Logout, user_id: 0005f910-15ed-d53e-089c-95ed40c8c7c9
2024.07.02 10:05:04.002058 [ 53667 ] {} AsynchronousMetrics: MemoryTracking: was 2.96 GiB, peak 141.72 GiB, free memory in arenas 152.01 MiB, will set to 2.96 GiB (RSS), difference: 3.14 MiB
2024.07.02 10:05:05.268413 [ 53523 ] {} Connection (10.74.58.30:9087): Connecting. Database: (not specified). User: ck_admin
**2024.07.02 10:05:05.268786 [ 53523 ] {} vip_order_analysis.dist_vip_user_group_order_analysis_test.DirectoryMonitor.default: Code: 210. DB::NetException: Connection refused (10.74.58.30:9087): While sending /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard15_replica1/34.bin. (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):**

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c630397 in /usr/bin/clickhouse
1. DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String const&) @ 0x0000000013046da8 in /usr/bin/clickhouse
2. DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x0000000013046762 in /usr/bin/clickhouse
3. DB::ConnectionPool::get(DB::ConnectionTimeouts const&, DB::Settings const*, bool) @ 0x000000000c7c54be in /usr/bin/clickhouse
4. DB::DistributedAsyncInsertDirectoryQueue::processFile(String&) @ 0x0000000012adfbb6 in /usr/bin/clickhouse
5. DB::DistributedAsyncInsertDirectoryQueue::processFiles() @ 0x0000000012ad9669 in /usr/bin/clickhouse
6. void std::__function::__policy_invoker::__call_impl const&, String const&, std::shared_ptr, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000012ae3468 in /usr/bin/clickhouse
7. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110af33f in /usr/bin/clickhouse
8. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110b0371 in /usr/bin/clickhouse
9. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c711464 in /usr/bin/clickhouse
10. start_thread @ 0x0000000000007ea5 in /usr/lib64/libpthread-2.17.so
11. __clone @ 0x00000000000fe96d in /usr/lib64/libc-2.17.so
(version 23.8.3.48 (official build))
2024.07.02 10:05:05.510703 [ 53471 ] {} DNSResolver: Updating DNS cache
2024.07.02 10:05:05.510967 [ 53471 ] {} DNSResolver: Updated DNS cache
2024.07.02 10:05:06.954556 [ 62180 ] {} HTTPHandler-factory: HTTP Request for HTTPHandler-factory. Method: GET, Address: 10.74.58.23:43766, User-Agent: Go-http-client/1.1, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2024.07.02 10:05:07.119883 [ 48024 ] {} HTTPHandler-factory: HTTP Request for HTTPHandler-factory. Method: GET, Address: 10.74.58.33:13694, User-Agent: Go-http-client/1.1, Content Type: , Transfer Encoding: identity, X-Forwarded-For: (none)
2024.07.02 10:05:09.691799 [ 53618 ] {} SystemLog (system.query_log): Flushing system log, 4 entries to flush up to offset 12289179
2024.07.02 10:05:09.694058 [ 53618 ] {} system.query_log (93922b5d-9728-42c0-9635-eb972b219e67): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2024.07.02 10:05:09.694085 [ 53618 ] {} DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 36.53 TiB.
2024.07.02 10:05:09.695301 [ 53618 ] {} MergedBlockOutputStream: filled checksums 202407_331491_331491_0 (state Temporary)
2024.07.02 10:05:09.695821 [ 53618 ] {} system.query_log (93922b5d-9728-42c0-9635-eb972b219e67): Renaming temporary part tmp_insert_202407_331491_331491_0 to 202407_331491_331491_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).

```
**2024.07.02 10:05:05.268786 [ 53523 ] {} vip_order_analysis.dist_vip_user_group_order_analysis_test.DirectoryMonitor.default: Code: 210. DB::NetException: Connection refused (10.74.58.30:9087): While sending /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard15_replica1/34.bin. (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):**

These abnormal behaviors can be seen in the logs. In theory, there should be no more requests for 58.30 for it to be correct. Currently, I have created a new distributed table with the same structure. After swapping the names of the new and old distributed tables, writing data through the new distributed table is working fine. However, the old distributed table still has the issue.

another info:
```
10.74.58.32 :) select * from distribution_queue;

SELECT *
FROM distribution_queue

Query id: 6422057a-5b7e-42d8-bed0-d2033545cc91

┌─database───────────┬─table───────────────────────────────────┬─data_path─────────────────────────────────────────────────────────────────────┬─is_blocked─┬─error_count─┬─data_files─┬─data_compressed_bytes─┬─broken_data_files─┬─broken_data_compressed_bytes─┬─last_exception────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─last_exception_time─┐
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard18_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard20_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard16_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard14_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard13_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard12_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard11_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard8_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard10_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard7_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard6_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard3_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard5_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard4_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard15_replica1/ │ 0 │ 13 │ 79 │ 47129289 │ 0 │ 0 │ Code: 210. DB::NetException: Connection refused (10.74.58.30:9087): While sending /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard15_replica1/9158.bin. (NETWORK_ERROR) (version 23.8.3.48 (official build)) │ 2024-07-02 12:00:29 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard1_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard9_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard19_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_bak │ /data/ck9087/store/11c/11c08f7b-a782-427c-abfa-11f9e298d805/shard2_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard18_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard20_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard16_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard14_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard13_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard12_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard11_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard8_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard10_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard7_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard6_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard3_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard5_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard4_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard15_replica1/ │ 0 │ 10 │ 8 │ 5075469 │ 0 │ 0 │ Code: 210. DB::NetException: Connection refused (10.74.58.30:9087): While sending /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard15_replica1/34.bin. (NETWORK_ERROR) (version 23.8.3.48 (official build)) │ 2024-07-02 12:00:35 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard1_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard9_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard19_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
│ vip_order_analysis │ dist_vip_user_group_order_analysis_test │ /data/ck9087/store/871/8718a828-1264-4ade-88ad-e7d9e723aef2/shard2_replica1/ │ 0 │ 0 │ 0 │ 0 │ 0 │ 0 │ │ 1970-01-01 08:00:00 │
└────────────────────┴─────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────┴────────────┴─────────────┴────────────┴───────────────────────┴───────────────────┴──────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────┘

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.