apache / apache/hudi

[SUPPORT] hudi.timeline.service.RequestHandler java.lang.OutOfMemoryError: Java heap space

Open
#10,075 1 comment 0 reactions 0 assignees View on GitHub
component:timeline-server priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

A clear and concise description of the problem.

**To Reproduce**

Steps to reproduce the behavior:

1.hudi 0.14

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :0.14

* Spark version :3.2.1

* Hive version :3.1.3

* Hadoop version :3.2.2

* Storage (HDFS/S3/GCS..) :cos

* Running on Docker? (yes/no) :no

**Additional context**

Add any other context about the problem here.

**Stacktrace**
```
23/11/13 16:53:10 ERROR Javalin: Exception occurred while servicing http-request
java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1005)
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89)
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58)
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.hudi.org.apache.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.apache.hudi.org.apache.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.apache.hudi.org.apache.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.apache.hudi.org.apache.jetty.server.Server.handle(Server.java:516)
at org.apache.hudi.org.apache.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.apache.hudi.org.apache.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.apache.hudi.org.apache.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.apache.hudi.org.apache.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.apache.hudi.org.apache.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.apache.hudi.org.apache.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.apache.hudi.org.apache.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.apache.hudi.org.apache.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:465)
at com.fasterxml.jackson.core.io.SegmentedStringWriter.getAndClear(SegmentedStringWriter.java:85)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3748)
at org.apache.hudi.timeline.service.RequestHandler.jsonifyResult(RequestHandler.java:121)
at org.apache.hudi.timeline.service.RequestHandler.writeValueAsStringSync(RequestHandler.java:212)
at org.apache.hudi.timeline.service.RequestHandler.writeValueAsString(RequestHandler.java:207)
at org.apache.hudi.timeline.service.RequestHandler.lambda$registerFileSlicesAPI$51(RequestHandler.java:409)
at org.apache.hudi.timeline.service.RequestHandler$$Lambda$4119/2033041916.handle(Unknown Source)
at org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:550)
at io.javalin.core.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:20)
at io.javalin.core.JavalinConfig$Inner$$Lambda$4086/1367226395.manage(Unknown Source)
at io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96)
at io.javalin.http.JavalinServlet$$Lambda$4100/702954435.handle(Unknown Source)
at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:99)
at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85)
at io.javalin.http.JavalinServletHandler$$Lambda$4501/56741594.apply(Unknown Source)
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89)
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58)
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.hudi.org.apache.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
23/11/13 16:53:20 ERROR Javalin: Exception occurred while servicing http-request
java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1005)
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89)
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58)
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.hudi.org.apache.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.apache.hudi.org.apache.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.apache.hudi.org.apache.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.apache.hudi.org.apache.jetty.server.Server.handle(Server.java:516)
at org.apache.hudi.org.apache.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.apache.hudi.org.apache.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.apache.hudi.org.apache.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.apache.hudi.org.apache.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.apache.hudi.org.apache.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.apache.hudi.org.apache.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.apache.hudi.org.apache.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.apache.hudi.org.apache.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with `RequestHandler.java`, especially `jsonifyResult` and the `registerFileSlicesAPI` handler identified in the stack trace. Investigate what response data is serialized by that endpoint and how the reported heap exhaustion can be reproduced; the issue provides no reproduction steps beyond using Hudi 0.14. Done means addressing the out-of-memory failure and validating the behavior with an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.