apache / apache/iotdb

[Bug] Msg: 305 executeStatement failed. Fetch Schema failed.

Abierto
#8,762 10 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Java
Estrellas
6.4k
Forks
1.2k
Merge medio
1 d 23 h
PR fusionados (30 d)
115

Descripción

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

version: 1.0.0

IoTDB> show cluster details;
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort| RpcAddress|RpcPort|DataConsensusPort|SchemaConsensusPort|MppPort|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
| 0|ConfigNode|Running| 127.0.0.1| 22277| 22278| | | | | |
| 1|ConfigNode|Running| 127.0.0.1| 32277| 32278| | | | | |
| 2|ConfigNode|Running| 127.0.0.1| 42277| 42278| | | | | |
| 3| DataNode|Running| 127.0.0.1| 9003| |192.168.4.165| 6667| 40010| 50010| 8777|
| 4| DataNode|Running| 127.0.0.1| 9004| |192.168.4.165| 6668| 40011| 50011| 8778|
| 5| DataNode|Running| 127.0.0.1| 9005| |192.168.4.165| 6669| 40012| 50012| 8779|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
Total line number = 6
It costs 0.021s

### Describe the bug and provide the minimal reproduce step

1. 连续写入大量数据
2. 写完后立即断电(模拟前段时间,某云服务器厂商大面积故障,导致我司业务瘫痪)
3. 重启后发现:数据量大的数据库,无法CRUD;数据量小的数据库正常
4. 原本以为大数据量需要时间恢复,等待24h后,问题依旧存在

以下是报错的详细日志:2023-01-05 18:14:05,983 [pool-314-IoTDB-ClientRPC-Processor-3] WARN o.a.i.d.u.ErrorHandlingUtils:62 - Status code: INTERNAL_SERVER_ERROR(305), operation: "select * from root.`1`.hardwareMonitor.0ed1ca81537c4ed88e64b15a13bae3e7". executeStatement failed
java.lang.RuntimeException: Fetch Schema failed.
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:202)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:126)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:98)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:76)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:208)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:161)
at org.apache.iotdb.db.mpp.plan.statement.crud.QueryStatement.accept(QueryStatement.java:507)
at org.apache.iotdb.db.mpp.plan.statement.StatementVisitor.process(StatementVisitor.java:107)
at org.apache.iotdb.db.mpp.plan.analyze.Analyzer.analyze(Analyzer.java:43)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.analyze(QueryExecution.java:258)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.(QueryExecution.java:142)
at org.apache.iotdb.db.mpp.plan.Coordinator.createQueryExecution(Coordinator.java:104)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:137)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementInternal(ClientRPCServiceImpl.java:215)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementV2(ClientRPCServiceImpl.java:390)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3453)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3433)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.iotdb.commons.exception.IoTDBException: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.dealWithException(QueryExecution.java:430)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getResult(QueryExecution.java:413)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getBatchResult(QueryExecution.java:439)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:200)
... 22 common frames omitted
Caused by: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.getActivatedSchemaTemplate(Traverser.java:471)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processOneLevelWildcard(Traverser.java:349)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:180)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:147)
at org.apache.iotdb.db.metadata.mtree.MTreeBelowSGMemoryImpl.fetchSchema(MTreeBelowSGMemoryImpl.java:768)
at org.apache.iotdb.db.metadata.schemaregion.SchemaRegionMemoryImpl.fetchSchema(SchemaRegionMemoryImpl.java:1148)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.fetchSchema(SchemaFetchScanOperator.java:116)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.next(SchemaFetchScanOperator.java:90)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchMergeOperator.next(SchemaFetchMergeOperator.java:76)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:186)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver$$Lambda$784/0x0000000000000000.get(Unknown Source)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:64)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:74)
2023-01-05 18:14:05,984 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 1
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:06,987 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 2
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:07,991 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 3
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted

### What did you expect to see?

1. 对于有问题的数据库,能恢复CRUD功能

### What did you see instead?

1. 允许损失数据,但CRUD功能正常

### Anything else?

定位问题期间,发现几个官网命令报错:
IoTDB> create snapshot for schema;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:7 no viable alternative at input 'create snapshot'
IoTDB> show variables;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:5 no viable alternative at input 'show variables'

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza con la ruta de error en ClusterSchemaFetcher.java y los frames de la pila relacionados con templates en Traverser.java y MTreeBelowSGMemoryImpl.java. Reproduce el escenario de reinicio después de una escritura intensiva descrito en el issue y, a continuación, rastrea por qué la obtención del esquema informa de que no hay ningún template montado. Se considera terminado cuando las consultas CRUD funcionan después del reinicio sin el fallo Fetch Schema.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
databases
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
38/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.