apache / apache/iotdb

[Bug] Msg: 305 executeStatement failed. Fetch Schema failed.

Open
#8,762 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.4k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
115

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

version: 1.0.0

IoTDB> show cluster details;
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort| RpcAddress|RpcPort|DataConsensusPort|SchemaConsensusPort|MppPort|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
| 0|ConfigNode|Running| 127.0.0.1| 22277| 22278| | | | | |
| 1|ConfigNode|Running| 127.0.0.1| 32277| 32278| | | | | |
| 2|ConfigNode|Running| 127.0.0.1| 42277| 42278| | | | | |
| 3| DataNode|Running| 127.0.0.1| 9003| |192.168.4.165| 6667| 40010| 50010| 8777|
| 4| DataNode|Running| 127.0.0.1| 9004| |192.168.4.165| 6668| 40011| 50011| 8778|
| 5| DataNode|Running| 127.0.0.1| 9005| |192.168.4.165| 6669| 40012| 50012| 8779|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
Total line number = 6
It costs 0.021s

### Describe the bug and provide the minimal reproduce step

1. 连续写入大量数据
2. 写完后立即断电(模拟前段时间,某云服务器厂商大面积故障,导致我司业务瘫痪)
3. 重启后发现:数据量大的数据库,无法CRUD;数据量小的数据库正常
4. 原本以为大数据量需要时间恢复,等待24h后,问题依旧存在

以下是报错的详细日志:2023-01-05 18:14:05,983 [pool-314-IoTDB-ClientRPC-Processor-3] WARN o.a.i.d.u.ErrorHandlingUtils:62 - Status code: INTERNAL_SERVER_ERROR(305), operation: "select * from root.`1`.hardwareMonitor.0ed1ca81537c4ed88e64b15a13bae3e7". executeStatement failed
java.lang.RuntimeException: Fetch Schema failed.
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:202)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:126)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:98)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:76)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:208)
at org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitQuery(AnalyzeVisitor.java:161)
at org.apache.iotdb.db.mpp.plan.statement.crud.QueryStatement.accept(QueryStatement.java:507)
at org.apache.iotdb.db.mpp.plan.statement.StatementVisitor.process(StatementVisitor.java:107)
at org.apache.iotdb.db.mpp.plan.analyze.Analyzer.analyze(Analyzer.java:43)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.analyze(QueryExecution.java:258)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.(QueryExecution.java:142)
at org.apache.iotdb.db.mpp.plan.Coordinator.createQueryExecution(Coordinator.java:104)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:137)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementInternal(ClientRPCServiceImpl.java:215)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementV2(ClientRPCServiceImpl.java:390)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3453)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatementV2.getResult(IClientRPCService.java:3433)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.iotdb.commons.exception.IoTDBException: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.dealWithException(QueryExecution.java:430)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getResult(QueryExecution.java:413)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getBatchResult(QueryExecution.java:439)
at org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:200)
... 22 common frames omitted
Caused by: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: There should not be no template mounted on any ancestor of a node usingTemplate.
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.getActivatedSchemaTemplate(Traverser.java:471)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processOneLevelWildcard(Traverser.java:349)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:180)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.processNameMatch(Traverser.java:402)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:182)
at org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.traverse(CollectorTraverser.java:63)
at org.apache.iotdb.db.metadata.mtree.traverser.Traverser.traverse(Traverser.java:147)
at org.apache.iotdb.db.metadata.mtree.MTreeBelowSGMemoryImpl.fetchSchema(MTreeBelowSGMemoryImpl.java:768)
at org.apache.iotdb.db.metadata.schemaregion.SchemaRegionMemoryImpl.fetchSchema(SchemaRegionMemoryImpl.java:1148)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.fetchSchema(SchemaFetchScanOperator.java:116)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchScanOperator.next(SchemaFetchScanOperator.java:90)
at org.apache.iotdb.db.mpp.execution.operator.schema.SchemaFetchMergeOperator.next(SchemaFetchMergeOperator.java:76)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:186)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver$$Lambda$784/0x0000000000000000.get(Unknown Source)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:64)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:74)
2023-01-05 18:14:05,984 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 1
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:06,987 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 2
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
2023-01-05 18:14:07,991 [pool-25-IoTDB-mpp-data-exchange-task-executors-6$20230105_101405_00068_3.0.0.0] WARN o.a.i.d.m.e.e.SourceHandle$GetDataBlocksTask:424 - failed to get data block [0, 1), attempt times: 3
org.apache.thrift.TException: Error in calling method getDataBlock, because: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.getDataBlock()
at org.apache.iotdb.db.mpp.execution.exchange.SourceHandle$GetDataBlocksTask.run(SourceHandle.java:402)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_getDataBlock, because: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.recv_getDataBlock()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.getDataBlock(MPPDataExchangeService.java:60)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$getDataBlock$6()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 7 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: Internal error processing getDataBlock
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Client.recv_getDataBlock(MPPDataExchangeService.java:73)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$recv_getDataBlock$8()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 13 common frames omitted
Caused by: org.apache.thrift.TApplicationException: Internal error processing getDataBlock
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a.CGLIB$receiveBase$22()
at org.apache.iotdb.commons.client.sync.SyncDataNodeMPPDataExchangeServiceClient$$EnhancerByCGLIB$$d254228a$$FastClassByCGLIB$$59c62a1.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted

### What did you expect to see?

1. 对于有问题的数据库,能恢复CRUD功能

### What did you see instead?

1. 允许损失数据,但CRUD功能正常

### Anything else?

定位问题期间,发现几个官网命令报错:
IoTDB> create snapshot for schema;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:7 no viable alternative at input 'create snapshot'
IoTDB> show variables;
Msg: 700: Error occurred while parsing SQL to physical plan: line 1:5 no viable alternative at input 'show variables'

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the failure path in ClusterSchemaFetcher.java and the template-related stack frames in Traverser.java and MTreeBelowSGMemoryImpl.java. Reproduce the restart-after-heavy-write scenario from the issue, then trace why schema fetching reports that no template is mounted. Done means CRUD queries work after restart without the Fetch Schema failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.