apache / apache/iotdb

[Bug] A large number of TimeSeries creations may trigger OOM, causing termination of client connections and inter-node communication threads.

Ouverte
#11,122 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Java
Étoiles
6.4k
Forks
1.2k
Merge moyen
1 j 23 h
PR mergées (30 j)
115

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

OS: CentOS 7.9
IoTDB: 1.1

### Describe the bug and provide the minimal reproduce step

Creating 19,400,000 TimeSeries using 5 processes in Session mode.

### What did you expect to see?

During stress testing, certain requests may be rejected if the load is too high.
After the stress testing, memory is freed up and performance is restored.

### What did you see instead?

During stress testing, when the on-heap memory and off-heap memory are fully utilized, there is a certain probability of encountering OutOfMemoryError in the error logs. This can lead to issues such as failed client connections, inability to synchronize data from other nodes to the problematic node, and failed execution of write statements.
![image](https://github.com/apache/iotdb/assets/24268919/95d94faa-6f2a-4aa9-9c9e-0a36c919843d)
![image](https://github.com/apache/iotdb/assets/24268919/e2ecee0c-58ba-4ca7-8b91-f98886ee31e3)

OOM in error log:
```
2023-09-05 15:17:16,306 [AsyncDataNodeInternalServiceClientPool-selector-174] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread AsyncDataNodeInternalServiceClientPool-selector-174-174
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.thrift.async.TAsyncMethodCall.doReadingResponseSize(TAsyncMethodCall.java:250)
at org.apache.thrift.async.TAsyncMethodCall.transition(TAsyncMethodCall.java:198)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.transitionMethods(TAsyncClientManager.java:143)
at org.apache.thrift.async.TAsyncClientManager$SelectThread.run(TAsyncClientManager.java:113)
2023-09-05 16:03:26,507 [ClientRPC-Service] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread ClientRPC-Service-177
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at java.io.BufferedOutputStream.(BufferedOutputStream.java:59)
at org.apache.thrift.transport.TSocket.(TSocket.java:86)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:138)
at org.apache.thrift.transport.TServerSocket.accept(TServerSocket.java:35)
at org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:153)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:139)
at org.apache.iotdb.commons.service.AbstractThriftServiceThread.run(AbstractThriftServiceThread.java:258)
2023-09-05 16:31:16,598 [pool-869-IoTDB-LogDispatcher-DataRegion[564]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[564]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,596 [pool-865-IoTDB-LogDispatcher-DataRegion[563]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-865-IoTDB-LogDispatcher-DataRegion[563]-1-218240
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:27:22,021 [pool-866-IoTDB-Compaction_Schedule-root.tc1-564-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-885-IoTDB-LogDispatcher-DataRegion[573]-1-234457
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-1] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,595 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:16,592 [pool-35-IoTDB-instance-management-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-47-IoTDB-DataNodeInternalRPC-Processor-458] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-458-238478
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:31:30,269 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:44:00,430 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [pool-8-IoTDB-TTL-Check-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:155 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:41,195 [3@group-000200000241-StateMachineUpdater] ERROR o.a.i.c.r.ApplicationStateMachineProxy:167 - application statemachine throws a runtime exception:
java.lang.OutOfMemoryError: Java heap space
2023-09-05 16:49:52,897 [pool-47-IoTDB-DataNodeInternalRPC-Processor-445] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-47-IoTDB-DataNodeInternalRPC-Processor-445-238350
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:06:40,158 [3@group-0002000000E3-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E3-FollowerState-238447
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:25:22,680 [pool-47-IoTDB-DataNodeInternalRPC-Processor-442] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:26:12,479 [pool-886-IoTDB-Compaction_Schedule-root.tc2-574-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:18:11,641 [3@group-0002000000E5-FollowerState] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread 3@group-0002000000E5-FollowerState-238443
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,415 [pool-890-IoTDB-Compaction_Schedule-root.tc2-576-1] ERROR o.a.i.d.e.s.DataRegion:2168 - Meet error in compaction schedule.
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,414 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=ClientRPC-Processor
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:20:06,577 [pool-862-IoTDB-Compaction_Schedule-root.tc1-563-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179 - Schedule task failed
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-885-IoTDB-LogDispatcher-DataRegion[573]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[573]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,413 [pool-893-IoTDB-LogDispatcher-DataRegion[576]-2] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=LogDispatcher-DataRegion[576]
java.lang.OutOfMemoryError: Java heap space
2023-09-05 17:33:54,423 [pool-48-IoTDB-ClientRPC-Processor-4209] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread pool-48-IoTDB-ClientRPC-Processor-4209-234260
java.lang.OutOfMemoryError: Java heap space
```

The on-heap memory has not been released for a long time:
![image](https://github.com/apache/iotdb/assets/24268919/ed1fd1ad-9739-4d3c-8a63-b93b40580476)

Frequent Full GCs:
![image](https://github.com/apache/iotdb/assets/24268919/e90f8985-056a-4850-800f-dc55b412f9ec)

There are performance issues even if OOM does not occur, such as slow client connections, queries, and write operations, or even program becoming unresponsive until timeout.

### Anything else?

Currently, regarding the issue of OOM causing client connection failure, it has been verified through Arthas that when OOM is thrown in Socket, it is not properly handled in AbstractThriftServiceThread. The execution of the close method in the finally block leads to the closure of the thread pool.
![image](https://github.com/apache/iotdb/assets/24268919/99b6f6a1-444d-412f-b6dc-d7b2e512e477)

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par reproduire le cas de stress 19,400,000-TimeSeries dans IoTDB 1.1 à l’aide de cinq processus en mode Session et examinez les chemins OutOfMemoryError signalés. Lisez AbstractThriftServiceThread et la pile du client Thrift asynchrone affichée dans les logs, puis retracez l’impact de OOM sur les connexions client et les communications inter-nœuds. Le travail est considéré comme terminé lorsque la surcharge est rejetée ou contenue, que la mémoire est récupérée après le test et que les opérations client, de requête, d’écriture et de cluster restent utilisables.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java
Domaine
databases, distributed-systems
Type d'issue
Bug
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.