apache / apache/gluten

[VL] OOM on TPCDS q67 and q95

Open
#1,359 14 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

**Describe the bug**
```
java.lang.RuntimeException: Error during calling Java code from native code: java.lang.UnsupportedOperationException: Not enough spark off-heap execution memory. Acquired: 8388608, granted: 0. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application.
at io.glutenproject.memory.alloc.GlutenManagedReservationListener.reserveOrThrow(GlutenManagedReservationListener.java:53)
at io.glutenproject.vectorized.ArrowOutIterator.nativeHasNext(Native Method)
at io.glutenproject.vectorized.ArrowOutIterator.hasNextInternal(ArrowOutIterator.java:48)
at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at io.glutenproject.backendsapi.glutendata.GlutenIteratorApi$$anon$3.hasNext(GlutenIteratorApi.scala:300)
at io.glutenproject.vectorized.CloseableColumnBatchIterator.hasNext(CloseableColumnBatchIterator.scala:41)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.GlutenColumnarShuffleWriter.internalWrite(GlutenColumnarShuffleWriter.scala:96)
at org.apache.spark.shuffle.GlutenColumnarShuffleWriter.write(GlutenColumnarShuffleWriter.scala:196)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:552)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1533)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:555)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

at io.glutenproject.vectorized.ArrowOutIterator.nativeHasNext(Native Method)
at io.glutenproject.vectorized.ArrowOutIterator.hasNextInternal(ArrowOutIterator.java:48)
at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at io.glutenproject.backendsapi.glutendata.GlutenIteratorApi$$anon$3.hasNext(GlutenIteratorApi.scala:300)
at io.glutenproject.vectorized.CloseableColumnBatchIterator.hasNext(CloseableColumnBatchIterator.scala:41)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.GlutenColumnarShuffleWriter.internalWrite(GlutenColumnarShuffleWriter.scala:96)
at org.apache.spark.shuffle.GlutenColumnarShuffleWriter.write(GlutenColumnarShuffleWriter.scala:196)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:552)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1533)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:555)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```

**To Reproduce**
spark version: 3.3.1
executor memory: 8g
executor cores: 8
executor memoryOverhead: 1g
offHeap.size: 24g

Contributor guide

Open the contributing guide

Research direction

Start by reproducing TPC-DS q67 and q95 with Spark 3.3.1 and the listed executor and off-heap settings. Read GlutenManagedReservationListener.java, ArrowOutIterator.java, and GlutenColumnarShuffleWriter.scala in the reported stack trace to trace the failed reservation. Done means the queries no longer fail with the off-heap memory error under the reproduction configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.