apache / apache/gluten

OOM in HashBuild operator (ShuffledHashJoin)

Open
#9,674 6 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

Expected behavior: spill, not OOM
Actual behavior: OOM

Stacktrace
```
Job Aborted:Error: Job aborted.Job aborted due to stage failure: Task 42 in stage 5.0 failed 4 times, most recent failure: Lost task 42.3 in stage 5.0 (TID 9024) (100.67.5.65 executor 263): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::isBlocked failed for [operator: HashBuild, plan node ID: 2]: Error during calling Java code from native code: org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=2.5 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=1280.0 MiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=640.0 MiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
Task.9024: Current used bytes: 2.5 GiB, peak bytes: N/A
\- Gluten.Tree.0: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
\- root.0: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
+- NativePlanEvaluator-0.0: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| \- single: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| +- root: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| | +- task.Gluten_Stage_5_TID_9024_VTID_0: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| | | +- node.2: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| | | | +- op.2.1.0.HashBuild: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| | | | \- op.2.0.0.HashProbe: Current used bytes: 4.5 MiB, peak bytes: 13.0 MiB
| | | +- node.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.0.0.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.3: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.3.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.5: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.5.0.0.TopN: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.4: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.4.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- node.1: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.1.1.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- ArrowContextInstance.0: Current used bytes: 8.0 MiB, peak bytes: 8.0 MiB
+- CelebornShuffleReader.0: Current used bytes: 1024.0 KiB, peak bytes: 10.0 MiB
| \- single: Current used bytes: 1024.0 KiB, peak bytes: 10.0 MiB
| +- root: Current used bytes: 21.3 KiB, peak bytes: 3.0 MiB
| | \- default_leaf: Current used bytes: 21.3 KiB, peak bytes: 2.3 MiB
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- CelebornShuffleReader.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 3.0 MiB
+- CelebornShuffleWriter.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- IndicatorVectorBase#init.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- CelebornShuffleWriter.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- IteratorMetrics.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- NativePlanEvaluator-0.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 698.4 MiB
+- IteratorMetrics.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- IndicatorVectorBase#init.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B

at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:49)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43)
at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.shuffle.VeloxCelebornColumnarShuffleWriter.internalWrite(VeloxCelebornColumnarShuffleWriter.scala:68)
at org.apache.spark.shuffle.CelebornColumnarShuffleWriter.write(CelebornColumnarShuffleWriter.scala:119)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:137)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1471)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```

### Gluten version

main branch

### Spark version

Spark-3.2.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with the HashBuild failure and the memory path named in the stack trace, including ThrowOnOomMemoryTarget.java, ManagedReservationListener.java, and ColumnarBatchOutIterator.hasNext0. Reproduce the reported Spark 3.2.x configuration if possible and trace the spill behavior; done means the ShuffledHashJoin spills instead of failing with an off-heap OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.