[Bug] Msg: 305 executeStatement failed. Fetch Schema failed.
- 主要言語
- Java
- スター
- 6.4k
- フォーク
- 1.2k
- 平均マージ
- 1日 23時間
- マージ済み PR(30日)
- 115
説明
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
version: 1.0.0
IoTDB> show cluster details;
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort| RpcAddress|RpcPort|DataConsensusPort|SchemaConsensusPort|MppPort|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
| 0|ConfigNode|Running| 127.0.0.1| 22277| 22278| | | | | |
| 1|ConfigNode|Running| 127.0.0.1| 32277| 32278| | | | | |
| 2|ConfigNode|Running| 127.0.0.1| 42277| 42278| | | | | |
| 3| DataNode|Running| 127.0.0.1| 9003| |192.168.4.165| 6667| 40010| 50010| 8777|
| 4| DataNode|Running| 127.0.0.1| 9004| |192.168.4.165| 6668| 40011| 50011| 8778|
| 5| DataNode|Running| 127.0.0.1| 9005| |192.168.4.165| 6669| 40012| 50012| 8779|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
Total line number = 6
It costs 0.021s
### Describe the bug and provide the minimal reproduce step
1. 连续写入大量数据
2. 写完后立即断电(模拟前段时间,某云服务器厂商大面积故障,导致我司业务瘫痪)
3. 重启后发现:数据量大的数据库,无法CRUD;数据量小的数据库正常
4. 原本以为大数据量需要时间恢复,等待24h后,问题依旧存在
以下是报错的详细日志:2023-01-05 18:14:05,983 [pool-314-IoTDB-ClientRPC-Processor-3] WARN o.a.i.d.u.ErrorHandlingUtils:62 - Status code: INTERNAL_SERVER_ERROR(305), operation: "select * from root.`1`.hardwareMonitor.0ed1ca81537c4ed88e64b15a13bae3e7". executeStatement failed
java.lang.RuntimeException: Fetch Schema failed.
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:202)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:126)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:98)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:76)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:208)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:161)
at org.apache.iotdb.db.mpp.plan.statement.crud.QueryStatement.accept(QueryStatement.java:507)
at org.apache.iotdb.db.mpp.plan.statement.StatementVisitor.process(StatementVisitor.java:107)
at org.apache.iotdb.db.mpp.plan.analyze.Analyzer.analyze(Analyzer.java:43)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.analyze(QueryExecution.java:258)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.(QueryExecution.java:142)
at org.apache.iotdb.db.mpp.plan.Coordinator.createQueryExecution(Coordinator.java:104)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:137)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementInternal(ClientRPCServiceImpl.java:215)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementV2(ClientRPCServiceImpl.java:390)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3453)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3433)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.iotdb.commons.exception.IoTDBException: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.dealWithException(QueryExecution.java:430)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getResult(QueryExecution.java:413)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getBatchResult(QueryExecution.java:439)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:200)
... 22 common frames omitted
Caused by: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.getActivatedSchemaTemplate(Traverser.java:471)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processOneLevelWildcard(Traverser.java:349)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:180)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:147)
at org.apache.iotdb.db.metadata.mtree.MTreeBelowSGMemoryImpl.fetchSchema(MTreeBelowSGMemoryImpl.java:768)
at org.apache.iotdb.db.metadata.schemaregion.SchemaRegionMemoryImpl.fetchSchema(SchemaRegionMemoryImpl.java:1148)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.fetchSchema(SchemaFetchScanOperator.java:116)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.next(SchemaFetchScanOperator.java:90)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchMergeOperator.next(SchemaFetchMergeOperator.java:76)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:186)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver$$Lambda$784/0x0000000000000000.get(Unknown Source)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:64)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:74)
2023-01-05 18:14:05,984 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 1
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:06,987 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 2
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:07,991 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 3
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
### What did you expect to see?
1. 对于有问题的数据库,能恢复CRUD功能
### What did you see instead?
1. 允许损失数据,但CRUD功能正常
### Anything else?
定位问题期间,发现几个官网命令报错:
IoTDB> create snapshot for schema;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:7 no viable alternative at input 'create snapshot'
IoTDB> show variables;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:5 no viable alternative at input 'show variables'
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
コントリビューションガイド
調査の方向性
ClusterSchemaFetcher.java の失敗パスと、Traverser.java および MTreeBelowSGMemoryImpl.java の template 関連のスタックフレームから始めてください。issue に記載されている、大量の書き込み後に再起動するシナリオを再現し、その後、スキーマの取得で template がマウントされていないと報告される理由を追跡してください。再起動後に Fetch Schema の失敗なしで CRUD クエリが動作すれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java
- 領域
- databases
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 静か
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 38/100