[Bug] Msg: 305 executeStatement failed. Fetch Schema failed.
- Vorherrschende Sprache
- Java
- Sterne
- 6.4k
- Forks
- 1.2k
- Ø Merge
- 1 T. 23 Std.
- Gemergte PRs (30 T.)
- 115
Beschreibung
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
version: 1.0.0
IoTDB> show cluster details;
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort| RpcAddress|RpcPort|DataConsensusPort|SchemaConsensusPort|MppPort|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
| 0|ConfigNode|Running| 127.0.0.1| 22277| 22278| | | | | |
| 1|ConfigNode|Running| 127.0.0.1| 32277| 32278| | | | | |
| 2|ConfigNode|Running| 127.0.0.1| 42277| 42278| | | | | |
| 3| DataNode|Running| 127.0.0.1| 9003| |192.168.4.165| 6667| 40010| 50010| 8777|
| 4| DataNode|Running| 127.0.0.1| 9004| |192.168.4.165| 6668| 40011| 50011| 8778|
| 5| DataNode|Running| 127.0.0.1| 9005| |192.168.4.165| 6669| 40012| 50012| 8779|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
Total line number = 6
It costs 0.021s
### Describe the bug and provide the minimal reproduce step
1. 连续写入大量数据
2. 写完后立即断电(模拟前段时间,某云服务器厂商大面积故障,导致我司业务瘫痪)
3. 重启后发现:数据量大的数据库,无法CRUD;数据量小的数据库正常
4. 原本以为大数据量需要时间恢复,等待24h后,问题依旧存在
以下是报错的详细日志:2023-01-05 18:14:05,983 [pool-314-IoTDB-ClientRPC-Processor-3] WARN o.a.i.d.u.ErrorHandlingUtils:62 - Status code: INTERNAL_SERVER_ERROR(305), operation: "select * from root.`1`.hardwareMonitor.0ed1ca81537c4ed88e64b15a13bae3e7". executeStatement failed
java.lang.RuntimeException: Fetch Schema failed.
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:202)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:126)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:98)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:76)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:208)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:161)
at org.apache.iotdb.db.mpp.plan.statement.crud.QueryStatement.accept(QueryStatement.java:507)
at org.apache.iotdb.db.mpp.plan.statement.StatementVisitor.process(StatementVisitor.java:107)
at org.apache.iotdb.db.mpp.plan.analyze.Analyzer.analyze(Analyzer.java:43)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.analyze(QueryExecution.java:258)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.(QueryExecution.java:142)
at org.apache.iotdb.db.mpp.plan.Coordinator.createQueryExecution(Coordinator.java:104)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:137)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementInternal(ClientRPCServiceImpl.java:215)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementV2(ClientRPCServiceImpl.java:390)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3453)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3433)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.iotdb.commons.exception.IoTDBException: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.dealWithException(QueryExecution.java:430)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getResult(QueryExecution.java:413)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getBatchResult(QueryExecution.java:439)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:200)
... 22 common frames omitted
Caused by: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.getActivatedSchemaTemplate(Traverser.java:471)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processOneLevelWildcard(Traverser.java:349)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:180)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:147)
at org.apache.iotdb.db.metadata.mtree.MTreeBelowSGMemoryImpl.fetchSchema(MTreeBelowSGMemoryImpl.java:768)
at org.apache.iotdb.db.metadata.schemaregion.SchemaRegionMemoryImpl.fetchSchema(SchemaRegionMemoryImpl.java:1148)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.fetchSchema(SchemaFetchScanOperator.java:116)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.next(SchemaFetchScanOperator.java:90)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchMergeOperator.next(SchemaFetchMergeOperator.java:76)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:186)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver$$Lambda$784/0x0000000000000000.get(Unknown Source)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:64)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:74)
2023-01-05 18:14:05,984 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 1
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:06,987 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 2
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:07,991 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 3
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
### What did you expect to see?
1. 对于有问题的数据库,能恢复CRUD功能
### What did you see instead?
1. 允许损失数据,但CRUD功能正常
### Anything else?
定位问题期间,发现几个官网命令报错:
IoTDB> create snapshot for schema;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:7 no viable alternative at input 'create snapshot'
IoTDB> show variables;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:5 no viable alternative at input 'show variables'
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Beitragsleitfaden
Rechercherichtung
Beginne mit dem Fehlerpfad in ClusterSchemaFetcher.java und den templatebezogenen Stackframes in Traverser.java und MTreeBelowSGMemoryImpl.java. Reproduziere das im Issue beschriebene Szenario eines Neustarts nach umfangreichen Schreibvorgängen und verfolge anschließend, warum das Abrufen des Schemas meldet, dass kein Template gemountet ist. Als abgeschlossen gilt die Aufgabe, wenn CRUD-Abfragen nach dem Neustart ohne den Fehler Fetch Schema funktionieren.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java
- Bereich
- databases
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 38/100