ioTDB运行一段时间报这个错误是什么原因 org.apache.iotdb.db.exception.BatchProcessException: Batch process failed:[TSStatus(code:411, message:Batch process failed:[TSStatus(code:413)])
- Dominant language
- Java
- Stars
- 6.4k
- Forks
- 1.2k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 115
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
0.13.3
### Describe the bug and provide the minimal reproduce step
---
### What did you expect to see?
运行一段时间后出现以下异常:
2022-12-07 05:14:00,167 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.f.MemTableFlushTask:187 - Storage group root.tb/0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 5ms
2022-12-07 05:14:00,170 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.w.n.ExclusiveWriteLogNode:273 - Log node root.tb/0-1670357048721-73-0-0.tsfile cleaned old file
2022-12-07 05:14:00,170 [pool-6-IoTDB-Flush-6] WARN o.a.i.d.rescon.SystemInfo:152 - SG (root.tb) released memory (delta: -13237666), but system is still in reject status (totalSgMemCost: 832284344).
2022-12-07 05:14:00,172 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.TsFileProcessor:1206 - The compression ratio of tsfile /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile is 85.33333333333333, totalMemTableSize: 53760, the file size: 630
2022-12-07 05:14:00,173 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 78.80125284761064
2022-12-07 05:14:00,174 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.TsFileProcessor:1224 - Start to end file file is /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, status: UNCLOSED
2022-12-07 05:14:00,175 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.TsFileProcessor:1228 - Ended file file is /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, status: UNCLOSED
2022-12-07 05:14:00,182 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.VirtualStorageGroupProcessor:2214 - signal closing storage group condition in root.tb-0
2022-12-07 05:14:00,183 [pool-6-IoTDB-Flush-6] WARN o.a.i.d.rescon.SystemInfo:152 - SG (root.tb) released memory (delta: 1896), but system is still in reject status (totalSgMemCost: 832282448).
2022-12-07 05:14:00,183 [pool-6-IoTDB-Flush-6] INFO o.a.i.d.e.s.TsFileProcessor:1242 - Storage group root.tb/0 close the file /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, TsFile size is 1605, time consumption of flushing metadata is 9ms
2022-12-07 05:14:05,816 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] WARN o.a.i.d.e.c.c.r.s.RewriteCompactionFileSelector:322 - /iotdb/sbin/../data/data/sequence/root.tb/0/0/1669887999251-7-0-61.tsfile is selected for cross space compaction, it is overlapped with /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile. It's selected because its end time 1670097600000 is greater than or equals to unseq file's endTime 1669852800000 in device root.tb.3ab8290000bd11ed9409d76c0e04bbb5
2022-12-07 05:14:05,816 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] WARN o.a.i.d.e.c.c.r.s.RewriteCompactionFileSelector:322 - /iotdb/sbin/../data/data/sequence/root.tb/0/0/1669887999251-7-0-61.tsfile is selected for cross space compaction, it is overlapped with /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile. It's selected because its end time 1670097600000 is greater than or equals to unseq file's endTime 1669852800000 in device root.tb.4034092000eb11edbe82a163009b23f6
2022-12-07 05:14:05,825 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] INFO o.a.i.d.e.c.c.r.s.RewriteCompactionFileSelector:136 - Selected merge candidates, 1 seqFiles, 1 unseqFiles, total memory cost 1473960, time consumption 10ms
2022-12-07 05:14:05,825 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] INFO o.a.i.d.e.c.c.r.RewriteCrossSpaceCompactionSelector:116 - select files for cross compaction, sequence files: [file is /iotdb/sbin/../data/data/sequence/root.tb/0/0/1669887999251-7-0-61.tsfile, status: CLOSED], unsequence files [file is /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, status: CLOSED], memory cost is 1473960
2022-12-07 05:14:05,826 [pool-20-IoTDB-Compaction_Schedule-root.tb-0-1] INFO o.a.i.d.e.c.c.r.RewriteCrossSpaceCompactionSelector:133 - root.tb-0 [Compaction] submit a task with 1 sequence file and 1 unseq files
2022-12-07 05:14:05,827 [pool-9-IoTDB-Compaction-8] INFO o.a.i.d.e.c.c.r.t.RewriteCrossSpaceCompactionTask:149 - root.tb-0 [Compaction] CrossSpaceCompactionTask start. Sequence files : [file is /iotdb/sbin/../data/data/sequence/root.tb/0/0/1669887999251-7-0-61.tsfile, status: COMPACTING], unsequence files : [file is /iotdb/sbin/../data/data/unsequence/root.tb/0/0/1670357048721-73-0-0.tsfile, status: COMPACTING], seq files size is 64.20209980010986 MB, unseq file size is 0.0015306472778320312 MB, total size is 64.2036304473877 MB
2022-12-07 05:14:10,257 [pool-17-IoTDB-RPC-Client-11] WARN o.a.i.d.u.ErrorHandlingUtils:146 - Exception occurred while processing non-query.
org.apache.iotdb.db.exception.BatchProcessException: Batch process failed:[TSStatus(code:411, message:Batch process failed:[TSStatus(code:413)])]
at org.apache.iotdb.db.qp.executor.PlanExecutor.insertTabletSerial(PlanExecutor.java:1936)
at org.apache.iotdb.db.qp.executor.PlanExecutor.insertTablet(PlanExecutor.java:1916)
at org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:306)
at org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:54)
at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:2202)
at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.insertTabletsInternally(TSServiceImpl.java:1699)
at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.insertTablets(TSServiceImpl.java:1649)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablets.getResult(TSIService.java:3626)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablets.getResult(TSIService.java:3606)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
### What did you see instead?
---
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start with PlanExecutor.java around insertTabletSerial (line 1936), then trace status codes 411 and 413 through the insert path. Review the related SystemInfo memory warnings and compaction log entries, and reproduce the long-running insert scenario on IoTDB 0.13.3. Done means the cause of the BatchProcessException and an applicable resolution are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100