apache / apache/iotdb

[Bug] A large number of TimeSeries creations may trigger OOM, causing termination of client connections and inter-node communication threads.

Offen
#11,122 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
6.4k
Forks
1.2k
Ø Merge
1 T. 23 Std.
Gemergte PRs (30 T.)
115

Beschreibung

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

OS: CentOS 7.9
IoTDB: 1.1

### Describe the bug and provide the minimal reproduce step

Creating 19,400,000 TimeSeries using 5 processes in Session mode.

### What did you expect to see?

During stress testing, certain requests may be rejected if the load is too high.
After the stress testing, memory is freed up and performance is restored.

### What did you see instead?

During stress testing, when the on-heap memory and off-heap memory are fully utilized, there is a certain probability of encountering OutOfMemoryError in the error logs. This can lead to issues such as failed client connections, inability to synchronize data from other nodes to the problematic node, and failed execution of write statements.
![image](https://github.com/apache/iotdb/assets/24268919/95d94faa-6f2a-4aa9-9c9e-0a36c919843d)
![image](https://github.com/apache/iotdb/assets/24268919/e2ecee0c-58ba-4ca7-8b91-f98886ee31e3)

OOM in error log:
```
2023-09-05 15:17:16,306 [AsyncDataNodeInternalServiceClientPool-selector-174] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread AsyncDataNodeInternalServiceClientPool-selector-174-174
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.thrift.async.TAsyncMethodCall.doReadingResponseSize(TAsyncMethodCall.java:250)
at org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:198)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.transitionMethods(TAsyncClientManager.java:143)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.run(TAsyncClientManager.java:113)
2023-09-05 16:03:26,507 [ClientRPC-Service] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread ClientRPC-Service-177
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at java.io.BufferedOutputStream.(BufferedOutputStream.java:59)
at org.apache.thrift.transport.TSocket.(TSocket.java:86)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:138)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:35)
at org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:153)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:139)
at org.apache.iotdb.commons.service.AbstractThriftServiceThread.run(AbstractThriftServiceThread.java:258)
2023-09-05 16:31:16,598 [pool-869-IoTDB-LogDispatcher-DataRegion[564]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[564]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,596 [pool-865-IoTDB-LogDispatcher-DataRegion[563]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-865-IoTDB-LogDispatcher-DataRegion[563]-1-218240
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:27:22,021 [pool-866-IoTDB-Compaction_Schedule-root.tc1-564-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-885-IoTDB-LogDispatcher-DataRegion[573]-1-234457
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-1] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,592 [pool-35-IoTDB-instance-management-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-47-IoTDB-DataNodeInternalRPC-Processor-458] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-458-238478
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:44:00,430 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [pool-8-IoTDB-TTL-Check-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:155 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:52,897 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-445-238350
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:06:40,158 [3@group-0002000000E3-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E3-FollowerState-238447
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:25:22,680 [pool-47-IoTDB-DataNodeInternalRPC-Processor-442] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:26:12,479 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:18:11,641 [3@group-0002000000E5-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E5-FollowerState-238443
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,415 [pool-890-IoTDB-Compaction_Schedule-root.tc2-576-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,414 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=ClientRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:20:06,577 [pool-862-IoTDB-Compaction_Schedule-root.tc1-563-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[573]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,423 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-48-IoTDB-ClientRPC-Processor-4209-234260
java.lang.OutOfMemoryError: Java heap space
```

The on-heap memory has not been released for a long time:
![image](https://github.com/apache/iotdb/assets/24268919/ed1fd1ad-9739-4d3c-8a63-b93b40580476)

Frequent Full GCs:
![image](https://github.com/apache/iotdb/assets/24268919/e90f8985-056a-4850-800f-dc55b412f9ec)

There are performance issues even if OOM does not occur, such as slow client connections, queries, and write operations, or even program becoming unresponsive until timeout.

### Anything else?

Currently, regarding the issue of OOM causing client connection failure, it has been verified through Arthas that when OOM is thrown in Socket, it is not properly handled in AbstractThriftServiceThread. The execution of the close method in the finally block leads to the closure of the thread pool.
![image](https://github.com/apache/iotdb/assets/24268919/99b6f6a1-444d-412f-b6dc-d7b2e512e477)

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, den Stressfall 19,400,000-TimeSeries in IoTDB 1.1 mithilfe von fünf Prozessen im Session-Modus zu reproduzieren und die gemeldeten OutOfMemoryError-Pfade zu untersuchen. Lies AbstractThriftServiceThread und den in den Logs gezeigten asynchronen Thrift-Client-Stack, und verfolge anschließend, wie sich OOM auf Clientverbindungen und die Kommunikation zwischen Knoten auswirkt. Als abgeschlossen gilt die Aufgabe, wenn Überlastung abgewiesen oder eingedämmt wird, sich der Speicher nach dem Test erholt und Client-, Abfrage-, Schreib- und Clusteroperationen weiterhin nutzbar sind.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
databases, distributed-systems
Issue-Typ
Bug
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.