[Bug] Msg: 305 executeStatement failed. Fetch Schema failed.
- Lenguaje dominante
- Java
- Estrellas
- 6.4k
- Forks
- 1.2k
- Merge medio
- 1 d 23 h
- PR fusionados (30 d)
- 115
Descripción
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
version: 1.0.0
IoTDB> show cluster details;
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort| RpcAddress|RpcPort|DataConsensusPort|SchemaConsensusPort|MppPort|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
| 0|ConfigNode|Running| 127.0.0.1| 22277| 22278| | | | | |
| 1|ConfigNode|Running| 127.0.0.1| 32277| 32278| | | | | |
| 2|ConfigNode|Running| 127.0.0.1| 42277| 42278| | | | | |
| 3| DataNode|Running| 127.0.0.1| 9003| |192.168.4.165| 6667| 40010| 50010| 8777|
| 4| DataNode|Running| 127.0.0.1| 9004| |192.168.4.165| 6668| 40011| 50011| 8778|
| 5| DataNode|Running| 127.0.0.1| 9005| |192.168.4.165| 6669| 40012| 50012| 8779|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
Total line number = 6
It costs 0.021s
### Describe the bug and provide the minimal reproduce step
1. 连续写入大量数据
2. 写完后立即断电(模拟前段时间,某云服务器厂商大面积故障,导致我司业务瘫痪)
3. 重启后发现:数据量大的数据库,无法CRUD;数据量小的数据库正常
4. 原本以为大数据量需要时间恢复,等待24h后,问题依旧存在
以下是报错的详细日志:2023-01-05 18:14:05,983 [pool-314-IoTDB-ClientRPC-Processor-3] WARN o.a.i.d.u.ErrorHandlingUtils:62 - Status code: INTERNAL_SERVER_ERROR(305), operation: "select * from root.`1`.hardwareMonitor.0ed1ca81537c4ed88e64b15a13bae3e7". executeStatement failed
java.lang.RuntimeException: Fetch Schema failed.
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:202)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:126)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:98)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:76)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:208)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:161)
at org.apache.iotdb.db.mpp.plan.statement.crud.QueryStatement.accept(QueryStatement.java:507)
at org.apache.iotdb.db.mpp.plan.statement.StatementVisitor.process(StatementVisitor.java:107)
at org.apache.iotdb.db.mpp.plan.analyze.Analyzer.analyze(Analyzer.java:43)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.analyze(QueryExecution.java:258)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.(QueryExecution.java:142)
at org.apache.iotdb.db.mpp.plan.Coordinator.createQueryExecution(Coordinator.java:104)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:137)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementInternal(ClientRPCServiceImpl.java:215)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementV2(ClientRPCServiceImpl.java:390)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3453)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3433)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.iotdb.commons.exception.IoTDBException: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.dealWithException(QueryExecution.java:430)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getResult(QueryExecution.java:413)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getBatchResult(QueryExecution.java:439)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:200)
... 22 common frames omitted
Caused by: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.getActivatedSchemaTemplate(Traverser.java:471)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processOneLevelWildcard(Traverser.java:349)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:180)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:147)
at org.apache.iotdb.db.metadata.mtree.MTreeBelowSGMemoryImpl.fetchSchema(MTreeBelowSGMemoryImpl.java:768)
at org.apache.iotdb.db.metadata.schemaregion.SchemaRegionMemoryImpl.fetchSchema(SchemaRegionMemoryImpl.java:1148)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.fetchSchema(SchemaFetchScanOperator.java:116)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.next(SchemaFetchScanOperator.java:90)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchMergeOperator.next(SchemaFetchMergeOperator.java:76)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:186)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver$$Lambda$784/0x0000000000000000.get(Unknown Source)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:64)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:74)
2023-01-05 18:14:05,984 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 1
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:06,987 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 2
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:07,991 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 3
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
### What did you expect to see?
1. 对于有问题的数据库,能恢复CRUD功能
### What did you see instead?
1. 允许损失数据,但CRUD功能正常
### Anything else?
定位问题期间,发现几个官网命令报错:
IoTDB> create snapshot for schema;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:7 no viable alternative at input 'create snapshot'
IoTDB> show variables;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:5 no viable alternative at input 'show variables'
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Guía de contribución
Línea de trabajo
Comienza con la ruta de error en ClusterSchemaFetcher.java y los frames de la pila relacionados con templates en Traverser.java y MTreeBelowSGMemoryImpl.java. Reproduce el escenario de reinicio después de una escritura intensiva descrito en el issue y, a continuación, rastrea por qué la obtención del esquema informa de que no hay ningún template montado. Se considera terminado cuando las consultas CRUD funcionan después del reinicio sin el fallo Fetch Schema.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- databases
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Tranquilo
- Claridad
- Bastante claro
- Aptitud para principiantes
- 38/100