[Bug] A large number of TimeSeries creations may trigger OOM, causing termination of client connections and inter-node communication threads.
- Lingua principale
- Java
- Stelle
- 6.4k
- Fork
- 1.2k
- Merge medio
- 1g 23h
- PR unite (30g)
- 115
Descrizione
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
OS: CentOS 7.9
IoTDB: 1.1
### Describe the bug and provide the minimal reproduce step
Creating 19,400,000 TimeSeries using 5 processes in Session mode.
### What did you expect to see?
During stress testing, certain requests may be rejected if the load is too high.
After the stress testing, memory is freed up and performance is restored.
### What did you see instead?
During stress testing, when the on-heap memory and off-heap memory are fully utilized, there is a certain probability of encountering OutOfMemoryError in the error logs. This can lead to issues such as failed client connections, inability to synchronize data from other nodes to the problematic node, and failed execution of write statements.


OOM in error log:
```
2023-09-05 15:17:16,306 [AsyncDataNodeInternalServiceClientPool-selector-174] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread AsyncDataNodeInternalServiceClientPool-selector-174-174
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.thrift.async.TAsyncMethodCall.doReadingResponseSize(TAsyncMethodCall.java:250)
at org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:198)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.transitionMethods(TAsyncClientManager.java:143)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.run(TAsyncClientManager.java:113)
2023-09-05 16:03:26,507 [ClientRPC-Service] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread ClientRPC-Service-177
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at java.io.BufferedOutputStream.(BufferedOutputStream.java:59)
at org.apache.thrift.transport.TSocket.(TSocket.java:86)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:138)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:35)
at org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:153)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:139)
at org.apache.iotdb.commons.service.AbstractThriftServiceThread.run(AbstractThriftServiceThread.java:258)
2023-09-05 16:31:16,598 [pool-869-IoTDB-LogDispatcher-DataRegion[564]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[564]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,596 [pool-865-IoTDB-LogDispatcher-DataRegion[563]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-865-IoTDB-LogDispatcher-DataRegion[563]-1-218240
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:27:22,021 [pool-866-IoTDB-Compaction_Schedule-root.tc1-564-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-885-IoTDB-LogDispatcher-DataRegion[573]-1-234457
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-1] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,592 [pool-35-IoTDB-instance-management-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-47-IoTDB-DataNodeInternalRPC-Processor-458] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-458-238478
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:44:00,430 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [pool-8-IoTDB-TTL-Check-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:155 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:52,897 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-445-238350
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:06:40,158 [3@group-0002000000E3-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E3-FollowerState-238447
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:25:22,680 [pool-47-IoTDB-DataNodeInternalRPC-Processor-442] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:26:12,479 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:18:11,641 [3@group-0002000000E5-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E5-FollowerState-238443
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,415 [pool-890-IoTDB-Compaction_Schedule-root.tc2-576-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,414 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=ClientRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:20:06,577 [pool-862-IoTDB-Compaction_Schedule-root.tc1-563-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[573]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,423 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-48-IoTDB-ClientRPC-Processor-4209-234260
java.lang.OutOfMemoryError: Java heap space
```
The on-heap memory has not been released for a long time:

Frequent Full GCs:

There are performance issues even if OOM does not occur, such as slow client connections, queries, and write operations, or even program becoming unresponsive until timeout.
### Anything else?
Currently, regarding the issue of OOM causing client connection failure, it has been verified through Arthas that when OOM is thrown in Socket, it is not properly handled in AbstractThriftServiceThread. The execution of the close method in the finally block leads to the closure of the thread pool.

### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia riproducendo il caso di stress 19,400,000-TimeSeries in IoTDB 1.1 usando cinque processi in modalità Session e analizza i percorsi di OutOfMemoryError segnalati. Leggi AbstractThriftServiceThread e lo stack del client Thrift asincrono mostrato nei log, quindi traccia in che modo OOM influisce sulle connessioni client e sulla comunicazione tra i nodi. Il lavoro è completo quando il sovraccarico viene rifiutato o contenuto, la memoria viene recuperata dopo il test e le operazioni client, di query, di scrittura e di cluster restano utilizzabili.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- java
- Ambito
- databases, distributed-systems
- Tipo di issue
- Bug
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 25/100