[Bug] can't connect to node TEndPoint。code:606, message:Reject write because there are too many requests need to process
- Dominant language
- Java
- Stars
- 6.4k
- Forks
- 1.2k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 115
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
IoTDB> show version;
+-------+-----------+
|Version| BuildInfo|
+-------+-----------+
| 1.0.0|fbbca3f-dev|
+-------+-----------+
Total line number = 1
It costs 0.002s
IoTDB> show cluster details;
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort| RpcAddress|RpcPort|DataConsensusPort|SchemaConsensusPort|MppPort|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
| 0|ConfigNode|Running| 127.0.0.1| 22277| 22278| | | | | |
| 1|ConfigNode|Running| 127.0.0.1| 32277| 32278| | | | | |
| 2|ConfigNode|Running| 127.0.0.1| 42277| 42278| | | | | |
| 3| DataNode|Running| 127.0.0.1| 9003| |192.168.4.165| 6667| 40010| 50010| 8777|
| 4| DataNode|Running| 127.0.0.1| 9004| |192.168.4.165| 6668| 40011| 50011| 8778|
| 5| DataNode|Running| 127.0.0.1| 9005| |192.168.4.165| 6669| 40012| 50012| 8779|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
Total line number = 6
It costs 0.005s
### Describe the bug and provide the minimal reproduce step
1. 使用Java客户端,一直连续插入大量数据
2. 一段时间后,插入线程卡死
3. 一段时间后,插入线程恢复
4. 持续运行中,以上2,3现象反复交替出现
5. 期间,查询正常,服务器节点报错如下:
DataNode0:
2023-01-10 17:00:04,076 [pool-27-IoTDB-ClientRPC-Processor-53$20230110_085944_28488_3.1.0] WARN o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:195 - can't connect to node TEndPoint(ip:127.0.0.1, port:9005)
org.apache.thrift.TException: Error in calling method sendPlanNode, because: Error in calling method recv_sendPlanNode, because: Error in calling method receiveBase, because: java.net.SocketTimeoutException: Read timed out
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.sendPlanNode()
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:171)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:139)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchWriteSync(FragmentInstanceDispatcherImpl.java:119)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatch(FragmentInstanceDispatcherImpl.java:90)
at org.apache.iotdb.db.mpp.plan.scheduler.ClusterScheduler.start(ClusterScheduler.java:106)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:288)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:206)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:150)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:164)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.insertRecords(ClientRPCServiceImpl.java:973)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertRecords.getResult(IClientRPCService.java:4128)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertRecords.getResult(IClientRPCService.java:4108)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_sendPlanNode, because: Error in calling method receiveBase, because: java.net.SocketTimeoutException: Read timed out
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.recv_sendPlanNode()
at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Client.sendPlanNode(IDataNodeRPCService.java:546)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.CGLIB$sendPlanNode$11()
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa$$FastClassByCGLIB$$a5fbc3b9.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: java.net.SocketTimeoutException: Read timed out
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Client.recv_sendPlanNode(IDataNodeRPCService.java:559)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.CGLIB$recv_sendPlanNode$13()
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa$$FastClassByCGLIB$$a5fbc3b9.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 25 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:112)
at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:107)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.CGLIB$receiveBase$174()
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa$$FastClassByCGLIB$$a5fbc3b9.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 31 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
... 43 common frames omitted
================================================================
DataNode2:
2023-01-10 17:36:36,991 [pool-27-IoTDB-ClientRPC-Processor-42$20230110_093606_28017_5.1.0] WARN o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:236 - write locally failed. TSStatus: TSStatus(code:606, message:Reject write because there are too many requests need to process), message: Reject write because there are too many requests need to process
### What did you expect to see?
1. 数据能一直持续正常插入,而不是一会正常一会卡死
### What did you see instead?
1. 数据插入时,若此时服务器不能正常插入,直接返回异常
### Anything else?
1. 每次停顿结束后,磁盘容量占用会减少
停顿中:
df -h
/dev/sda6 3.4T 569G 2.7T 18% /nfv
停顿结束:
df -h
/dev/sda6 3.4T 517G 2.7T 16% /nfv
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start with the Java client reproduction using continuous bulk inserts, then read FragmentInstanceDispatcherImpl.dispatchWriteSync and the local write rejection around code 606. Compare the timeout path in sendPlanNode with the reported rejection behavior. Done means sustained inserts no longer alternate between hanging and recovering, and an overloaded server returns an exception directly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100