[SUPPORT] OOM during a Sync/Async clean operation
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**OOM during a Sync or Async clean operation**
ENV:
Hudi version: 0.11.1
Java Version 1.8
Spark Version: 3.1.2
EMR version: 6.4
Clean Policy: KEEP_LATEST_BY_HOURS -- 24 hours(default)
Clean Parallelism: 200 (default)
Metadata: disabled
We have been experiencing consistent OOM errors when running Hudi delta-streamer job in continuous mode. The oom occurs during the "Generating list of file slices to be cleaned" phase. The image below shows the heap growth during the clean operation.
The heap growth particularly happens during 2 API calls from the executors getReplacedFileGroupsBefore and getAllFileGroups on the file system view.

Also, we do have jfr files that contain memory profiles during a failed clean operation, Github does not allow us to attach them to the issue.
We also tried the following setting `hoodie.embed.timeline.server.async: true` which seems to have reduced the heap usage and the clean succeeds. This seems to happen due to the single threaded nature of the async executor. Using this setting we notice the following heap usage. However, the clean operation is slightly less performant. We will continue to use this setting for now, and post any new observations on this ticket.

Flame Graph for the Async clean with async timeline server

Flame Graph for Async clean with sync timeline server

**Expected behavior**
A clean operation should not fail with OOM
**Environment Description**
* Hudi version : 0.11.1
* Spark version : 3.2.1
* Hadoop version : 3.2.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```qtp1729765409-406
at java.lang.OutOfMemoryError.()V (OutOfMemoryError.java:48)
at java.lang.StringCoding.encode(Ljava/nio/charset/Charset;[CII)[B (StringCoding.java:350)
at java.lang.String.getBytes(Ljava/nio/charset/Charset;)[B (String.java:941)
at io.javalin.Context.result(Ljava/lang/String;)Lio/javalin/Context; (Context.kt:364)
at org.apache.hudi.timeline.service.RequestHandler.writeValueAsStringSync(Lio/javalin/Context;Ljava/lang/Object;)V (RequestHandler.java:210)
at org.apache.hudi.timeline.service.RequestHandler.writeValueAsString(Lio/javalin/Context;Ljava/lang/Object;)V (RequestHandler.java:176)
at org.apache.hudi.timeline.service.RequestHandler.lambda$registerFileSlicesAPI$18(Lio/javalin/Context;)V (RequestHandler.java:384)
at org.apache.hudi.timeline.service.RequestHandler$$Lambda$2356.handle(Lio/javalin/Context;)V (Unknown Source)
at org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(Lio/javalin/Context;)V (RequestHandler.java:501)
at io.javalin.security.SecurityUtil.noopAccessManager(Lio/javalin/Handler;Lio/javalin/Context;Ljava/util/Set;)V (SecurityUtil.kt:22)
at io.javalin.Javalin$$Lambda$2336.manage(Lio/javalin/Handler;Lio/javalin/Context;Ljava/util/Set;)V (Unknown Source)
at io.javalin.Javalin.lambda$addHandler$0(Lio/javalin/Handler;Ljava/util/Set;Lio/javalin/Context;)V (Javalin.java:606)
at io.javalin.Javalin$$Lambda$2340.handle(Lio/javalin/Context;)V (Unknown Source)
at io.javalin.core.JavalinServlet$service$2$1.invoke()V (JavalinServlet.kt:46)
at io.javalin.core.JavalinServlet$service$2$1.invoke()Ljava/lang/Object; (JavalinServlet.kt:17)
at io.javalin.core.JavalinServlet$service$1.invoke(Lkotlin/jvm/functions/Function0;)V (JavalinServlet.kt:143)
at io.javalin.core.JavalinServlet$service$2.invoke()V (JavalinServlet.kt:41)
at io.javalin.core.JavalinServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (JavalinServlet.kt:107)
at io.javalin.core.util.JettyServerUtil$initialize$httpHandler$1.doHandle(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (JettyServerUtil.kt:72)
at org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.nextScope(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ScopedHandler.java:203)
at org.apache.hudi.org.eclipse.jetty.servlet.ServletHandler.doScope(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ServletHandler.java:480)
at org.apache.hudi.org.eclipse.jetty.server.session.SessionHandler.doScope(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (SessionHandler.java:1668)
at org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.nextScope(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ScopedHandler.java:201)
at org.apache.hudi.org.eclipse.jetty.server.handler.ContextHandler.doScope(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ContextHandler.java:1247)
at org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.handle(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ScopedHandler.java:144)
at org.apache.hudi.org.eclipse.jetty.server.handler.HandlerList.handle(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (HandlerList.java:61)
at org.apache.hudi.org.eclipse.jetty.server.handler.StatisticsHandler.handle(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (StatisticsHandler.java:174)
at org.apache.hudi.org.eclipse.jetty.server.handler.HandlerWrapper.handle(Ljava/lang/String;Lorg/apache/hudi/org/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (HandlerWrapper.java:132)
at org.apache.hudi.org.eclipse.jetty.server.Server.handle(Lorg/apache/hudi/org/eclipse/jetty/server/HttpChannel;)V (Server.java:502)
at org.apache.hudi.org.eclipse.jetty.server.HttpChannel.handle()Z (HttpChannel.java:370)
at org.apache.hudi.org.eclipse.jetty.server.HttpConnection.onFillable()V (HttpConnection.java:267)
at org.apache.hudi.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded()V (AbstractConnection.java:305)
at org.apache.hudi.org.eclipse.jetty.io.FillInterest.fillable()Z (FillInterest.java:103)
at org.apache.hudi.org.eclipse.jetty.io.ChannelEndPoint$2.run()V (ChannelEndPoint.java:117)
at org.apache.hudi.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(Ljava/lang/Runnable;)V (EatWhatYouKill.java:333)
at org.apache.hudi.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(Z)Z (EatWhatYouKill.java:310)
at org.apache.hudi.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(Z)V (EatWhatYouKill.java:168)
at org.apache.hudi.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run()V (EatWhatYouKill.java:126)
at org.apache.hudi.org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run()V (ReservedThreadExecutor.java:366)
at org.apache.hudi.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Ljava/lang/Runnable;)V (QueuedThreadPool.java:765)
at org.apache.hudi.org.eclipse.jetty.util.thread.QueuedThreadPool$2.run()V (QueuedThreadPool.java:683)
at java.lang.Thread.run()V (Thread.java:750)```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the clean phase's getReplacedFileGroupsBefore and getAllFileGroups calls into RequestHandler.java, especially registerFileSlicesAPI and writeValueAsStringSync, using the provided stack trace. Done means the reported Sync/Async clean operation no longer exhausts heap while generating file slices; validate against the stated Hudi, Spark, EMR, and S3 setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend-api-design, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100