ioTDB运行一段时间报这个错误是什么原因 org.apache.iotdb.db.exception.BatchProcessException: Batch process failed:[TSStatus(code:411, message:Batch process failed:[TSStatus(code:413)])
- Ngôn ngữ chính
- Java
- Star
- 6.4k
- Fork
- 1.2k
- Merge trung bình
- 1 ngày 23 giờ
- Pull request đã merge (30 ngày)
- 115
Mô tả
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
0.13.3
### Describe the bug and provide the minimal reproduce step
---
### What did you expect to see?
运行一段时间后出现以下异常:
2022-12-07 05:14:00,167 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.f.MemTableFlushTask:187 - Storage group root.tb/0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 5ms
2022-12-07 05:14:00,170 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.w.n.ExclusiveWriteLogNode:273 - Log node root.tb/0-1670357048721-73-0-0.tsfile cleaned old file
2022-12-07 05:14:00,170 [pool-6-IoTDB-Flush-6] WARN o.a.i.d.rescon.SystemInfo:152 - SG (root.tb) released memory (delta: -13237666), but system is still in reject status (totalSgMemCost: 832284344).
2022-12-07 05:14:00,172 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.TsFileProcessor:1206 - The compression ratio of tsfile /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile is 85.33333333333333, totalMemTableSize: 53760, the file size: 630
2022-12-07 05:14:00,173 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 78.80125284761064
2022-12-07 05:14:00,174 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.TsFileProcessor:1224 - Start to end file file is /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, status: UNCLOSED
2022-12-07 05:14:00,175 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.TsFileProcessor:1228 - Ended file file is /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, status: UNCLOSED
2022-12-07 05:14:00,182 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.VirtualStorageGroupProcessor:2214 - signal closing storage group condition in root.tb-0
2022-12-07 05:14:00,183 [pool-6-IoTDB-Flush-6] WARN o.a.i.d.rescon.SystemInfo:152 - SG (root.tb) released memory (delta: 1896), but system is still in reject status (totalSgMemCost: 832282448).
2022-12-07 05:14:00,183 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.TsFileProcessor:1242 - Storage group root.tb/0 close the file /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, TsFile size is 1605, time consumption of flushing metadata is 9ms
2022-12-07 05:14:05,816 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] WARN o.a.i.d.e.c.c.r.s.RewriteCompactionFileSelector:322 - /iotdb/sbin/../data/data/sequence/root.tb/0/0/1669887999251-7-0-61.tsfile is selected for cross space compaction, it is overlapped with /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile. It's selected because its end time 1670097600000 is greater than or equals to unseq file's endTime 1669852800000 in device root.tb.3ab8290000bd11ed9409d76c0e04bbb5
2022-12-07 05:14:05,816 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] WARN o.a.i.d.e.c.c.r.s.RewriteCompactionFileSelector:322 - /iotdb/sbin/../data/data/sequence/root.tb/0/0/1669887999251-7-0-61.tsfile is selected for cross space compaction, it is overlapped with /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile. It's selected because its end time 1670097600000 is greater than or equals to unseq file's endTime 1669852800000 in device root.tb.4034092000eb11edbe82a163009b23f6
2022-12-07 05:14:05,825 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] INFO o.a.i.d.e.c.c.r.s.RewriteCompactionFileSelector:136 - Selected merge candidates, 1 seqFiles, 1 unseqFiles, total memory cost 1473960, time consumption 10ms
2022-12-07 05:14:05,825 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] INFO o.a.i.d.e.c.c.r.RewriteCrossSpaceCompactionSelector:116 - select files for cross compaction, sequence files: [file is /iotdb/sbin/../data/data/sequence/root.tb/0/0/1669887999251-7-0-61.tsfile, status: CLOSED], unsequence files [file is /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, status: CLOSED], memory cost is 1473960
2022-12-07 05:14:05,826 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] INFO o.a.i.d.e.c.c.r.RewriteCrossSpaceCompactionSelector:133 - root.tb-0 [Compaction] submit a task with 1 sequence file and 1 unseq files
2022-12-07 05:14:05,827 [pool-9-IoTDB-Compaction-8] INFO o.a.i.d.e.c.c.r.t.RewriteCrossSpaceCompactionTask:149 - root.tb-0 [Compaction] CrossSpaceCompactionTask start. Sequence files : [file is /iotdb/sbin/../data/data/sequence/root.tb/0/0/1669887999251-7-0-61.tsfile, status: COMPACTING], unsequence files : [file is /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, status: COMPACTING], seq files size is 64.20209980010986 MB, unseq file size is 0.0015306472778320312 MB, total size is 64.2036304473877 MB
2022-12-07 05:14:10,257 [pool-17-IoTDB-RPC-Client-11] WARN o.a.i.d.u.ErrorHandlingUtils:146 - Exception occurred while processing non-query.
org.apache.iotdb.db.exception.BatchProcessException: Batch process failed:[TSStatus(code:411, message:Batch process failed:[TSStatus(code:413)])]
at org.apache.iotdb.db.qp.executor.PlanExecutor.insertTabletSerial(PlanExecutor.java:1936)
at org.apache.iotdb.db.qp.executor.PlanExecutor.insertTablet(PlanExecutor.java:1916)
at org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:306)
at org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:54)
at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:2202)
at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.insertTabletsInternally(TSServiceImpl.java:1699)
at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.insertTablets(TSServiceImpl.java:1649)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablets.getResult(TSIService.java:3626)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablets.getResult(TSIService.java:3606)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
### What did you see instead?
---
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu với PlanExecutor.java, quanh insertTabletSerial (dòng 1936), sau đó theo dõi các mã trạng thái 411 và 413 qua đường dẫn chèn. Xem lại các cảnh báo bộ nhớ liên quan của SystemInfo và các mục nhật ký compaction, đồng thời tái hiện kịch bản chèn chạy trong thời gian dài trên IoTDB 0.13.3. Hoàn tất khi nguyên nhân của BatchProcessException và một giải pháp áp dụng được đã được xác định.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- java
- Lĩnh vực
- databases
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Cần làm rõ
- Mức phù hợp với người mới
- 20/100