apache / apache/iotdb

[Bug] A large number of TimeSeries creations may trigger OOM, causing termination of client connections and inter-node communication threads.

Aperta
#11,122 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
6.4k
Fork
1.2k
Merge medio
1g 23h
PR unite (30g)
115

Descrizione

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

OS: CentOS 7.9
IoTDB: 1.1

### Describe the bug and provide the minimal reproduce step

Creating 19,400,000 TimeSeries using 5 processes in Session mode.

### What did you expect to see?

During stress testing, certain requests may be rejected if the load is too high.
After the stress testing, memory is freed up and performance is restored.

### What did you see instead?

During stress testing, when the on-heap memory and off-heap memory are fully utilized, there is a certain probability of encountering OutOfMemoryError in the error logs. This can lead to issues such as failed client connections, inability to synchronize data from other nodes to the problematic node, and failed execution of write statements.
![image](https://github.com/apache/iotdb/assets/24268919/95d94faa-6f2a-4aa9-9c9e-0a36c919843d)
![image](https://github.com/apache/iotdb/assets/24268919/e2ecee0c-58ba-4ca7-8b91-f98886ee31e3)

OOM in error log:
```
2023-09-05 15:17:16,306 [AsyncDataNodeInternalServiceClientPool-selector-174] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread AsyncDataNodeInternalServiceClientPool-selector-174-174
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.thrift.async.TAsyncMethodCall.doReadingResponseSize(TAsyncMethodCall.java:250)
at org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:198)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.transitionMethods(TAsyncClientManager.java:143)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.run(TAsyncClientManager.java:113)
2023-09-05 16:03:26,507 [ClientRPC-Service] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread ClientRPC-Service-177
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at java.io.BufferedOutputStream.(BufferedOutputStream.java:59)
at org.apache.thrift.transport.TSocket.(TSocket.java:86)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:138)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:35)
at org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:153)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:139)
at org.apache.iotdb.commons.service.AbstractThriftServiceThread.run(AbstractThriftServiceThread.java:258)
2023-09-05 16:31:16,598 [pool-869-IoTDB-LogDispatcher-DataRegion[564]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[564]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,596 [pool-865-IoTDB-LogDispatcher-DataRegion[563]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-865-IoTDB-LogDispatcher-DataRegion[563]-1-218240
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:27:22,021 [pool-866-IoTDB-Compaction_Schedule-root.tc1-564-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-885-IoTDB-LogDispatcher-DataRegion[573]-1-234457
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-1] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,592 [pool-35-IoTDB-instance-management-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-47-IoTDB-DataNodeInternalRPC-Processor-458] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-458-238478
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:44:00,430 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [pool-8-IoTDB-TTL-Check-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:155 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:52,897 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-445-238350
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:06:40,158 [3@group-0002000000E3-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E3-FollowerState-238447
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:25:22,680 [pool-47-IoTDB-DataNodeInternalRPC-Processor-442] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:26:12,479 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:18:11,641 [3@group-0002000000E5-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E5-FollowerState-238443
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,415 [pool-890-IoTDB-Compaction_Schedule-root.tc2-576-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,414 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=ClientRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:20:06,577 [pool-862-IoTDB-Compaction_Schedule-root.tc1-563-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[573]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,423 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-48-IoTDB-ClientRPC-Processor-4209-234260
java.lang.OutOfMemoryError: Java heap space
```

The on-heap memory has not been released for a long time:
![image](https://github.com/apache/iotdb/assets/24268919/ed1fd1ad-9739-4d3c-8a63-b93b40580476)

Frequent Full GCs:
![image](https://github.com/apache/iotdb/assets/24268919/e90f8985-056a-4850-800f-dc55b412f9ec)

There are performance issues even if OOM does not occur, such as slow client connections, queries, and write operations, or even program becoming unresponsive until timeout.

### Anything else?

Currently, regarding the issue of OOM causing client connection failure, it has been verified through Arthas that when OOM is thrown in Socket, it is not properly handled in AbstractThriftServiceThread. The execution of the close method in the finally block leads to the closure of the thread pool.
![image](https://github.com/apache/iotdb/assets/24268919/99b6f6a1-444d-412f-b6dc-d7b2e512e477)

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia riproducendo il caso di stress 19,400,000-TimeSeries in IoTDB 1.1 usando cinque processi in modalità Session e analizza i percorsi di OutOfMemoryError segnalati. Leggi AbstractThriftServiceThread e lo stack del client Thrift asincrono mostrato nei log, quindi traccia in che modo OOM influisce sulle connessioni client e sulla comunicazione tra i nodi. Il lavoro è completo quando il sovraccarico viene rifiutato o contenuto, la memoria viene recuperata dopo il test e le operazioni client, di query, di scrittura e di cluster restano utilizzabili.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
databases, distributed-systems
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.