[Bug] can't connect to node TEndPoint。code:606, message:Reject write because there are too many requests need to process
- Lenguaje dominante
- Java
- Estrellas
- 6.4k
- Forks
- 1.2k
- Merge medio
- 1 d 23 h
- PR fusionados (30 d)
- 115
Descripción
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
IoTDB> show version;
+-------+-----------+
|Version| BuildInfo|
+-------+-----------+
| 1.0.0|fbbca3f-dev|
+-------+-----------+
Total line number = 1
It costs 0.002s
IoTDB> show cluster details;
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort| RpcAddress|RpcPort|DataConsensusPort|SchemaConsensusPort|MppPort|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
| 0|ConfigNode|Running| 127.0.0.1| 22277| 22278| | | | | |
| 1|ConfigNode|Running| 127.0.0.1| 32277| 32278| | | | | |
| 2|ConfigNode|Running| 127.0.0.1| 42277| 42278| | | | | |
| 3| DataNode|Running| 127.0.0.1| 9003| |192.168.4.165| 6667| 40010| 50010| 8777|
| 4| DataNode|Running| 127.0.0.1| 9004| |192.168.4.165| 6668| 40011| 50011| 8778|
| 5| DataNode|Running| 127.0.0.1| 9005| |192.168.4.165| 6669| 40012| 50012| 8779|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
Total line number = 6
It costs 0.005s
### Describe the bug and provide the minimal reproduce step
1. 使用Java客户端,一直连续插入大量数据
2. 一段时间后,插入线程卡死
3. 一段时间后,插入线程恢复
4. 持续运行中,以上2,3现象反复交替出现
5. 期间,查询正常,服务器节点报错如下:
DataNode0:
2023-01-10 17:00:04,076 [pool-27-IoTDB-ClientRPC-Processor-53$20230110_085944_28488_3.1.0] WARN o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:195 - can't connect to node TEndPoint(ip:127.0.0.1, port:9005)
org.apache.thrift.TException: Error in calling method sendPlanNode, because: Error in calling method recv_sendPlanNode, because: Error in calling method receiveBase, because: java.net.SocketTimeoutException: Read timed out
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.sendPlanNode()
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:171)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:139)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchWriteSync(FragmentInstanceDispatcherImpl.java:119)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatch(FragmentInstanceDispatcherImpl.java:90)
at org.apache.iotdb.db.mpp.plan.scheduler.ClusterScheduler.start(ClusterScheduler.java:106)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:288)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:206)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:150)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:164)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.insertRecords(ClientRPCServiceImpl.java:973)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertRecords.getResult(IClientRPCService.java:4128)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertRecords.getResult(IClientRPCService.java:4108)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_sendPlanNode, because: Error in calling method receiveBase, because: java.net.SocketTimeoutException: Read timed out
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.recv_sendPlanNode()
at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Client.sendPlanNode(IDataNodeRPCService.java:546)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.CGLIB$sendPlanNode$11()
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa$$FastClassByCGLIB$$a5fbc3b9.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: java.net.SocketTimeoutException: Read timed out
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Client.recv_sendPlanNode(IDataNodeRPCService.java:559)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.CGLIB$recv_sendPlanNode$13()
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa$$FastClassByCGLIB$$a5fbc3b9.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 25 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:112)
at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:107)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.CGLIB$receiveBase$174()
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa$$FastClassByCGLIB$$a5fbc3b9.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 31 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
... 43 common frames omitted
================================================================
DataNode2:
2023-01-10 17:36:36,991 [pool-27-IoTDB-ClientRPC-Processor-42$20230110_093606_28017_5.1.0] WARN o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:236 - write locally failed. TSStatus: TSStatus(code:606, message:Reject write because there are too many requests need to process), message: Reject write because there are too many requests need to process
### What did you expect to see?
1. 数据能一直持续正常插入,而不是一会正常一会卡死
### What did you see instead?
1. 数据插入时,若此时服务器不能正常插入,直接返回异常
### Anything else?
1. 每次停顿结束后,磁盘容量占用会减少
停顿中:
df -h
/dev/sda6 3.4T 569G 2.7T 18% /nfv
停顿结束:
df -h
/dev/sda6 3.4T 517G 2.7T 16% /nfv
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Guía de contribución
Línea de trabajo
Comienza con la reproducción en el cliente Java usando inserciones bulk continuas; después, lee FragmentInstanceDispatcherImpl.dispatchWriteSync y el rechazo de escritura local relacionado con el código 606. Compara la ruta de timeout en sendPlanNode con el comportamiento de rechazo informado. Se considera terminado cuando las inserciones sostenidas ya no alternan entre quedarse bloqueadas y recuperarse, y un servidor sobrecargado devuelve una excepción directamente.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- databases
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 35/100