apache / apache/iotdb

关于0.13版本的TsFile 同步方式bug

Đang mở
#8,618 6 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Java
Star
6.4k
Fork
1.2k
Merge trung bình
1 ngày 23 giờ
Pull request đã merge (30 ngày)
115

Mô tả

0.13版本中,通过 tools\start-sync-client 的方式进行TsFile同步,这种方式未来是否还会继续提供??1.0中的Pipe、PipeSink方式是未来的支持方式吗??

0.13版本中,TsFile 同步发现bug,操作方式如下:

1、根据文档描述,配置了 发送端和接收端;
2、执行tools\start-sync-client ,发送端将数据(root.XXX存储组)同步到接收端;
3、接收端,通过delete STORAGE group root.XXX 方式,删除存储组;
4、发送端,再次执行 tools\start-sync-client,无法再次同步 数据到接收端。

发送端日志:

2022-12-27 10:47:43,118 [main] INFO o.a.i.d.s.c.SyncSenderDescriptor:82 - Start to read sync config file D:\apache-iotdb-0.13.0-all-bin\conf\iotdb-sync-client.properties
2022-12-27 10:47:43,154 [main] INFO o.a.i.d.c.IoTDBDescriptor:128 - Start to read config file file:D:/apache-iotdb-0.13.0-all-bin/conf/iotdb-engine.properties
2022-12-27 10:47:43,165 [main] INFO o.a.i.d.c.IoTDBDescriptor:1296 - allocateMemoryForRead = 1125226905
2022-12-27 10:47:43,165 [main] INFO o.a.i.d.c.IoTDBDescriptor:1297 - allocateMemoryForWrite = 1500302540
2022-12-27 10:47:43,166 [main] INFO o.a.i.d.c.IoTDBDescriptor:1298 - allocateMemoryForSchema = 375075635
2022-12-27 10:47:43,168 [main] INFO o.a.i.t.c.c.TSFileDescriptor:123 - try loading iotdb-engine.properties from D:\apache-iotdb-0.13.0-all-bin\conf\iotdb-engine.properties
2022-12-27 10:47:43,173 [main] INFO o.a.i.d.c.IoTDBDescriptor:260 - IoTDB enable memory control: true
2022-12-27 10:47:43,201 [Sync-Client] INFO o.a.i.d.c.IoTDBThreadPoolFactory:174 - new single scheduled thread pool: timedForceMLogThread
2022-12-27 10:47:43,223 [Sync-Client] INFO o.a.i.d.w.i.SingleFileLogReader:144 - open WAL file: mlog.bin size is 10086971
2022-12-27 10:47:43,850 [Sync-Client] INFO o.a.i.m.c.MetricConfigDescriptor:75 - Start to read config file D:\apache-iotdb-0.13.0-all-bin\conf\iotdb-metric.yml
2022-12-27 10:47:43,873 [Sync-Client] INFO o.a.i.d.c.IoTDBThreadPoolFactory:195 - new scheduled thread pool: sync-client-timer
2022-12-27 10:47:43,955 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:481 - Receiver has received schema successfully.
2022-12-27 10:47:43,956 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:265 - There are 1 data dirs to be synced.
2022-12-27 10:47:43,956 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:268 - Start to sync data in data dir D:\apache-iotdb-0.13.0-all-bin\data\data, the process is 1/1
2022-12-27 10:47:43,957 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.r.SyncSenderLogAnalyzer:89 - log file D:\apache-iotdb-0.13.0-all-bin\data\data\sync-sender\192.168.235.134_9555\sync.log doesn't exist.
2022-12-27 10:47:43,963 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.m.SyncFileManager:94 - Start to get current local files in data folder D:\apache-iotdb-0.13.0-all-bin\data\data
2022-12-27 10:47:43,965 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.m.SyncFileManager:175 - Start to get last local files from last local file info D:\apache-iotdb-0.13.0-all-bin\data\data\sync-sender\192.168.235.134_9555\last_local_files.txt
2022-12-27 10:47:43,976 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:543 - Sync process starts to transfer data of storage group root.rczz, it has 1 virtual storage groups.
2022-12-27 10:47:43,976 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:597 - There has no deleted files to be synced in storage group root.rczz
2022-12-27 10:47:43,977 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:626 - There has no new tsfiles to be synced in storage group root.rczz
2022-12-27 10:47:43,978 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:580 - Sync process finished the task to sync data of storage group root.rczz.
2022-12-27 10:47:43,980 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:287 - Finish to sync data in data dir D:\apache-iotdb-0.13.0-all-bin\data\data, the process is 1/1
2022-12-27 10:47:43,988 [pool-2-IoTDB-sync-client-timer-1] INFO o.a.i.d.s.s.t.SyncClient:299 - Sync process has finished.

接收端日志:
2022-12-27 10:47:43,886 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:98 - Start to sync with sender 192.168.235.1_8506306abea44db18da7f89dabf66265
2022-12-27 10:47:43,944 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:292 - Start to load metadata in sync process.
2022-12-27 10:47:43,944 [Sync] INFO o.a.i.d.w.i.SingleFileLogReader:144 - open WAL file: mlog.bin size is 0
2022-12-27 10:47:43,965 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:155 - Sync process started to receive data of storage group root.rczz
2022-12-27 10:47:43,977 [Sync] INFO o.a.i.d.s.r.t.SyncServiceImpl:336 - Sync process with sender 192.168.235.1_8506306abea44db18da7f89dabf66265 finished.
2022-12-27 10:47:44,003 [pool-25-IoTDB-Load-TsFile-1] INFO o.a.i.d.s.r.l.FileLoader:177 - Sync loading process for 192.168.235.1_8506306abea44db18da7f89dabf66265 has finished.
2022-12-27 10:48:21,873 [pool-3-IoTDB-timedQuerySqlCount-1] INFO o.a.i.d.s.b.QueryFrequencyRecorder:42 - Query count in current 1 minute 2

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu bằng cách tái hiện workflow 0.13 được liệt kê với tools\start-sync-client: sync root.XXX, xóa nó trên receiver và chạy đồng bộ hóa lại. Đọc log của sender và receiver cùng với iotdb-sync-client.properties và iotdb-engine.properties để truy vết lý do không có tệp nào được chọn. Hoàn tất khi xác nhận được hành vi đồng bộ hóa lặp lại như mong đợi hoặc cô lập được lỗi bằng một regression test hay một cách tái hiện rõ ràng.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
java
Lĩnh vực
databases, distributed-systems
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Đình trệ
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
42/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.