OOM in HashBuild operator (ShuffledHashJoin)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
Expected behavior: spill, not OOM
Actual behavior: OOM
Stacktrace
```
Job Aborted:Error: Job aborted.Job aborted due to stage failure: Task 42 in stage 5.0 failed 4 times, most recent failure: Lost task 42.3 in stage 5.0 (TID 9024) (100.67.5.65 executor 263): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::isBlocked failed for [operator: HashBuild, plan node ID: 2]: Error during calling Java code from native code: org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=2.5 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=1280.0 MiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=640.0 MiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
Task.9024: Current used bytes: 2.5 GiB, peak bytes: N/A
\- Gluten.Tree.0: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
\- root.0: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
+- NativePlanEvaluator-0.0: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| \- single: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| +- root: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| | +- task.Gluten_Stage_5_TID_9024_VTID_0: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| | | +- node.2: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| | | | +- op.2.1.0.HashBuild: Current used bytes: 2.5 GiB, peak bytes: 2.5 GiB
| | | | \- op.2.0.0.HashProbe: Current used bytes: 4.5 MiB, peak bytes: 13.0 MiB
| | | +- node.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.0.0.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.3: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.3.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.5: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.5.0.0.TopN: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.4: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.4.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- node.1: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.1.1.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- ArrowContextInstance.0: Current used bytes: 8.0 MiB, peak bytes: 8.0 MiB
+- CelebornShuffleReader.0: Current used bytes: 1024.0 KiB, peak bytes: 10.0 MiB
| \- single: Current used bytes: 1024.0 KiB, peak bytes: 10.0 MiB
| +- root: Current used bytes: 21.3 KiB, peak bytes: 3.0 MiB
| | \- default_leaf: Current used bytes: 21.3 KiB, peak bytes: 2.3 MiB
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- CelebornShuffleReader.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 3.0 MiB
+- CelebornShuffleWriter.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- IndicatorVectorBase#init.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- CelebornShuffleWriter.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- IteratorMetrics.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- NativePlanEvaluator-0.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 698.4 MiB
+- IteratorMetrics.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- IndicatorVectorBase#init.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:49)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43)
at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.shuffle.VeloxCelebornColumnarShuffleWriter.internalWrite(VeloxCelebornColumnarShuffleWriter.scala:68)
at org.apache.spark.shuffle.CelebornColumnarShuffleWriter.write(CelebornColumnarShuffleWriter.scala:119)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:137)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1471)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
### Gluten version
main branch
### Spark version
Spark-3.2.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start with the HashBuild failure and the memory path named in the stack trace, including ThrowOnOomMemoryTarget.java, ManagedReservationListener.java, and ColumnarBatchOutIterator.hasNext0. Reproduce the reported Spark 3.2.x configuration if possible and trace the spill behavior; done means the ShuffledHashJoin spills instead of failing with an off-heap OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100