[VL] OOM after union two huge table
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
After union two huge table, Gluten report an OOM.

### Spark version
None
### Spark configurations
spark.gluten.memory.offHeap.size.in.bytes=89.6 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=5.6 GiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=2.8 GiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
### System information
_No response_
### Relevant logs
```bash
25/03/26 21:15:16 [58] [Executor task launch worker for task 271.0 in stage 27.0 (TID 1263)] ERROR ManagedReservationListener : Error reserving memory from target
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 24.0 MiB, granted: 2.0 MiB. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=89.6 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=5.6 GiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=2.8 GiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
Task.1263: Current used bytes: 9.0 GiB, peak bytes: N/A
+- org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@7d50189c: Current used bytes: 8.9 GiB, peak bytes: N/A
\- Gluten.Tree.79: Current used bytes: 21.0 MiB, peak bytes: 91.2 MiB
\- root.79: Current used bytes: 21.0 MiB, peak bytes: 91.2 MiB
+- ShuffleReader.10: Current used bytes: 9.0 MiB, peak bytes: 16.0 MiB
| \- single: Current used bytes: 9.0 MiB, peak bytes: 16.0 MiB
| +- gluten::MemoryAllocator: Current used bytes: 1518.3 KiB, peak bytes: 3.0 MiB
| \- ShuffleReader_root: Current used bytes: 960.0 KiB, peak bytes: 1024.0 KiB
| \- ShuffleReader_default_leaf: Current used bytes: 960.0 KiB, peak bytes: 960.0 KiB
+- ArrowContextInstance.10: Current used bytes: 8.0 MiB, peak bytes: 8.0 MiB
+- ColumnarToRow.2: Current used bytes: 3.0 MiB, peak bytes: 8.0 MiB
| \- single: Current used bytes: 3.0 MiB, peak bytes: 8.0 MiB
| +- ColumnarToRow_root: Current used bytes: 3.0 MiB, peak bytes: 3.0 MiB
| | \- ColumnarToRow_default_leaf: Current used bytes: 3.0 MiB, peak bytes: 3.0 MiB
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- WholeStageIterator.72: Current used bytes: 1024.0 KiB, peak bytes: 32.0 MiB
| \- single: Current used bytes: 1024.0 KiB, peak bytes: 32.0 MiB
| +- WholeStageIterator_root: Current used bytes: 24.0 KiB, peak bytes: 28.0 MiB
| | +- task.Gluten_Stage_27_TID_1263: Current used bytes: 24.0 KiB, peak bytes: 28.0 MiB
| | | +- node.2: Current used bytes: 24.0 KiB, peak bytes: 28.0 MiB
| | | | +- op.2.0.0.HashProbe: Current used bytes: 24.0 KiB, peak bytes: 2.6 MiB
| | | | \- op.2.1.0.HashBuild: Current used bytes: 0.0 B, peak bytes: 68.0 KiB
| | | +- node.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.0.0.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.4: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.4.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.3: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.3.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- node.1: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.1.1.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- WholeStageIterator_default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- OverAcquire.DummyTarget.171: Current used bytes: 0.0 B, peak bytes: 9.6 MiB
+- OverAcquire.DummyTarget.184: Current used bytes: 0.0 B, peak bytes: 2.4 MiB
+- OverAcquire.DummyTarget.167: Current used bytes: 0.0 B, peak bytes: 2.4 MiB
+- OverAcquire.DummyTarget.144: Current used bytes: 0.0 B, peak bytes: 4.8 MiB
+- BuildSideRelation#BatchSerializer.14: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
| \- single: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
| +- BuildSideRelation#BatchSerializer_root: Current used bytes: 0.0 B, peak bytes: 1024.0 KiB
| | \- BuildSideRelation#BatchSerializer_default_leaf: Current used bytes: 0.0 B, peak bytes: 2.3 KiB
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- ArrowContextInstance.15: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
at org.apache.gluten.memory.nmm.ManagedReservationListener.reserve(ManagedReservationListener.java:43)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:64)
at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.utils.InvocationFlowProtection.hasNext(Iterators.scala:156)
at org.apache.gluten.utils.IteratorCompleter.hasNext(Iterators.scala:68)
at org.apache.gluten.utils.PayloadCloser.hasNext(Iterators.scala:34)
at org.apache.gluten.utils.LifeTimeAccumulator.hasNext(Iterators.scala:97)
at org.apache.gluten.execution.VeloxColumnarToRowExec$$anon$1.hasNext(VeloxColumnarToRowExec.scala:131)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.TraversableOnce$FlattenOps$$anon$2.hasNext(TraversableOnce.scala:521)
```
Contributor guide
Assessment
This issue has not been assessed yet.