[Bug] A large number of TimeSeries creations may trigger OOM, causing termination of client connections and inter-node communication threads.
- 主要言語
- Java
- スター
- 6.4k
- フォーク
- 1.2k
- 平均マージ
- 1日 23時間
- マージ済み PR(30日)
- 115
説明
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
OS: CentOS 7.9
IoTDB: 1.1
### Describe the bug and provide the minimal reproduce step
Creating 19,400,000 TimeSeries using 5 processes in Session mode.
### What did you expect to see?
During stress testing, certain requests may be rejected if the load is too high.
After the stress testing, memory is freed up and performance is restored.
### What did you see instead?
During stress testing, when the on-heap memory and off-heap memory are fully utilized, there is a certain probability of encountering OutOfMemoryError in the error logs. This can lead to issues such as failed client connections, inability to synchronize data from other nodes to the problematic node, and failed execution of write statements.


OOM in error log:
```
2023-09-05 15:17:16,306 [AsyncDataNodeInternalServiceClientPool-selector-174] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread AsyncDataNodeInternalServiceClientPool-selector-174-174
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.thrift.async.TAsyncMethodCall.doReadingResponseSize(TAsyncMethodCall.java:250)
at org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:198)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.transitionMethods(TAsyncClientManager.java:143)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.run(TAsyncClientManager.java:113)
2023-09-05 16:03:26,507 [ClientRPC-Service] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread ClientRPC-Service-177
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at java.io.BufferedOutputStream.(BufferedOutputStream.java:59)
at org.apache.thrift.transport.TSocket.(TSocket.java:86)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:138)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:35)
at org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:153)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:139)
at org.apache.iotdb.commons.service.AbstractThriftServiceThread.run(AbstractThriftServiceThread.java:258)
2023-09-05 16:31:16,598 [pool-869-IoTDB-LogDispatcher-DataRegion[564]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[564]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,596 [pool-865-IoTDB-LogDispatcher-DataRegion[563]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-865-IoTDB-LogDispatcher-DataRegion[563]-1-218240
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:27:22,021 [pool-866-IoTDB-Compaction_Schedule-root.tc1-564-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-885-IoTDB-LogDispatcher-DataRegion[573]-1-234457
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-1] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,592 [pool-35-IoTDB-instance-management-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-47-IoTDB-DataNodeInternalRPC-Processor-458] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-458-238478
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:44:00,430 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [pool-8-IoTDB-TTL-Check-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:155 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:52,897 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-445-238350
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:06:40,158 [3@group-0002000000E3-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E3-FollowerState-238447
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:25:22,680 [pool-47-IoTDB-DataNodeInternalRPC-Processor-442] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:26:12,479 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:18:11,641 [3@group-0002000000E5-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E5-FollowerState-238443
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,415 [pool-890-IoTDB-Compaction_Schedule-root.tc2-576-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,414 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=ClientRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:20:06,577 [pool-862-IoTDB-Compaction_Schedule-root.tc1-563-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[573]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,423 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-48-IoTDB-ClientRPC-Processor-4209-234260
java.lang.OutOfMemoryError: Java heap space
```
The on-heap memory has not been released for a long time:

Frequent Full GCs:

There are performance issues even if OOM does not occur, such as slow client connections, queries, and write operations, or even program becoming unresponsive until timeout.
### Anything else?
Currently, regarding the issue of OOM causing client connection failure, it has been verified through Arthas that when OOM is thrown in Socket, it is not properly handled in AbstractThriftServiceThread. The execution of the close method in the finally block leads to the closure of the thread pool.

### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
コントリビューションガイド
調査の方向性
まず、5つのSessionモードプロセスを使用してIoTDB 1.1で19,400,000-TimeSeriesのストレスケースを再現し、報告されたOutOfMemoryErrorの発生経路を調査します。AbstractThriftServiceThreadとログに示された非同期Thriftクライアントスタックを読み、その後、OOMがクライアント接続とノード間通信にどのような影響を与えるかを追跡します。過負荷が拒否または抑制され、テスト後にメモリが回復し、クライアント、クエリ、書き込み、クラスタの各操作が引き続き利用可能であれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java
- 領域
- databases, distributed-systems
- issue の種類
- バグ
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 停滞
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 25/100