apache / apache/druid

OutOfMemory error while merging in AppenderatorImpl

Open
#6,338 3 comments 0 reactions 1 assignee Claimed by @jihoonson View on GitHub
Area - Batch Ingestion Area - Streaming Ingestion Improvement stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 31m
Merged PRs (30d)
209

Description

`AppenderatorImpl` currently merges all persisted indexes at once. This can incur OOM error like below if there are too many indexes to be merged when `longEncoding = auto` .

```
2018-09-13T23:33:55,073 ERROR [task-runner-0-priority-0] io.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
at io.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1042) ~[druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:455) [druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:457) [druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:429) [druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
at io.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:141) ~[druid-server-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:124) ~[druid-server-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1004) ~[druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
... 7 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at it.unimi.dsi.fastutil.longs.LongArrays.grow(LongArrays.java:222) ~[fastutil-8.1.0.jar:?]
at it.unimi.dsi.fastutil.longs.LongArrayList.grow(LongArrayList.java:277) ~[fastutil-8.1.0.jar:?]
at it.unimi.dsi.fastutil.longs.LongArrayList.add(LongArrayList.java:296) ~[fastutil-8.1.0.jar:?]
at io.druid.segment.data.IntermediateColumnarLongsSerializer.add(IntermediateColumnarLongsSerializer.java:89) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.LongColumnSerializer.serialize(LongColumnSerializer.java:87) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.IndexMergerV9.mergeIndexesAndWriteColumns(IndexMergerV9.java:458) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:209) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:837) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:710) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:688) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:662) ~[druid-server-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$0(AppenderatorImpl.java:566) ~[druid-server-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT]
at io.druid.segment.realtime.appenderator.AppenderatorImpl$$Lambda$117/371981722.apply(Unknown Source) ~[?:?]
at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
... 3 more
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.