apache / apache/iotdb

[Bug] can't connect to node TEndPoint。code:606, message:Reject write because there are too many requests need to process

オープン
#8,814 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Java
スター
6.4k
フォーク
1.2k
平均マージ
1日 23時間
マージ済み PR(30日)
115

説明

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

IoTDB> show version;
+-------+-----------+
|Version| BuildInfo|
+-------+-----------+
| 1.0.0|fbbca3f-dev|
+-------+-----------+
Total line number = 1
It costs 0.002s
IoTDB> show cluster details;
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort| RpcAddress|RpcPort|DataConsensusPort|SchemaConsensusPort|MppPort|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
| 0|ConfigNode|Running| 127.0.0.1| 22277| 22278| | | | | |
| 1|ConfigNode|Running| 127.0.0.1| 32277| 32278| | | | | |
| 2|ConfigNode|Running| 127.0.0.1| 42277| 42278| | | | | |
| 3| DataNode|Running| 127.0.0.1| 9003| |192.168.4.165| 6667| 40010| 50010| 8777|
| 4| DataNode|Running| 127.0.0.1| 9004| |192.168.4.165| 6668| 40011| 50011| 8778|
| 5| DataNode|Running| 127.0.0.1| 9005| |192.168.4.165| 6669| 40012| 50012| 8779|
+------+----------+-------+---------------+------------+-------------------+-------------+-------+-----------------+-------------------+-------+
Total line number = 6
It costs 0.005s

### Describe the bug and provide the minimal reproduce step

1. 使用Java客户端,一直连续插入大量数据
2. 一段时间后,插入线程卡死
3. 一段时间后,插入线程恢复
4. 持续运行中,以上2,3现象反复交替出现
5. 期间,查询正常,服务器节点报错如下:

DataNode0:
2023-01-10 17:00:04,076 [pool-27-IoTDB-ClientRPC-Processor-53$20230110_085944_28488_3.1.0] WARN o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:195 - can't connect to node TEndPoint(ip:127.0.0.1, port:9005)
org.apache.thrift.TException: Error in calling method sendPlanNode, because: Error in calling method recv_sendPlanNode, because: Error in calling method receiveBase, because: java.net.SocketTimeoutException: Read timed out
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.sendPlanNode()
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:171)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:139)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchWriteSync(FragmentInstanceDispatcherImpl.java:119)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatch(FragmentInstanceDispatcherImpl.java:90)
at org.apache.iotdb.db.mpp.plan.scheduler.ClusterScheduler.start(ClusterScheduler.java:106)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:288)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:206)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:150)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:164)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.insertRecords(ClientRPCServiceImpl.java:973)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertRecords.getResult(IClientRPCService.java:4128)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertRecords.getResult(IClientRPCService.java:4108)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.thrift.TException: Error in calling method recv_sendPlanNode, because: Error in calling method receiveBase, because: java.net.SocketTimeoutException: Read timed out
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.recv_sendPlanNode()
at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Client.sendPlanNode(IDataNodeRPCService.java:546)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.CGLIB$sendPlanNode$11()
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa$$FastClassByCGLIB$$a5fbc3b9.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 19 common frames omitted
Caused by: org.apache.thrift.TException: Error in calling method receiveBase, because: java.net.SocketTimeoutException: Read timed out
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:95)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.receiveBase()
at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Client.recv_sendPlanNode(IDataNodeRPCService.java:559)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.CGLIB$recv_sendPlanNode$13()
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa$$FastClassByCGLIB$$a5fbc3b9.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 25 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:112)
at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:107)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa.CGLIB$receiveBase$174()
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$2ab862aa$$FastClassByCGLIB$$a5fbc3b9.invoke()
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
... 31 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
... 43 common frames omitted

================================================================

DataNode2:
2023-01-10 17:36:36,991 [pool-27-IoTDB-ClientRPC-Processor-42$20230110_093606_28017_5.1.0] WARN o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:236 - write locally failed. TSStatus: TSStatus(code:606, message:Reject write because there are too many requests need to process), message: Reject write because there are too many requests need to process

### What did you expect to see?

1. 数据能一直持续正常插入,而不是一会正常一会卡死

### What did you see instead?

1. 数据插入时,若此时服务器不能正常插入,直接返回异常

### Anything else?
1. 每次停顿结束后,磁盘容量占用会减少
停顿中:
df -h
/dev/sda6 3.4T 569G 2.7T 18% /nfv
停顿结束:
df -h
/dev/sda6 3.4T 517G 2.7T 16% /nfv

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

まず、連続したバルク挿入を使って Java クライアントで再現し、次に FragmentInstanceDispatcherImpl.dispatchWriteSync と、コード 606 に関連するローカルな書き込み拒否を確認します。sendPlanNode のタイムアウト経路を、報告された拒否動作と比較します。連続した挿入がハングと復旧を交互に繰り返さなくなり、過負荷状態のサーバーが直接例外を返すようになれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
databases
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。