apache / apache/gluten

[VL][1.5] Not enough spark off-heap execution memory on RowToVeloxColumnar

Open
#11,541 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

error log:

```
26/02/01 15:22:11 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 2)
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 2000.0 MiB, granted: 1047.0 MiB. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=2.0 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=2.0 GiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=1024.0 MiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
Task.2: Current used bytes: 1001.0 MiB, peak bytes: N/A
\- Gluten.Tree.1: Current used bytes: 1001.0 MiB, peak bytes: 2.0 GiB
\- Capacity[8.0 EiB].1: Current used bytes: 1001.0 MiB, peak bytes: 2.0 GiB
+- ArrowContextInstance.1: Current used bytes: 1000.0 MiB, peak bytes: 2047.0 MiB
+- NativePlanEvaluator-1.0: Current used bytes: 1024.0 KiB, peak bytes: 8.0 MiB
| \- single: Current used bytes: 1024.0 KiB, peak bytes: 8.0 MiB
| +- root: Current used bytes: 1344.0 B, peak bytes: 1024.0 KiB
| | +- default_leaf: Current used bytes: 1344.0 B, peak bytes: 1472.0 B
| | \- task.Gluten_Stage_2_TID_2_VTID_1: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | +- node.6: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.6.0.0.Unnest: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | +- node.8: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.8.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | +- node.2: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.2.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | +- node.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.0.0.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | +- node.9: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.9.0.0.PartialAggregation: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | +- node.10: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.10.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | +- node.5: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.5.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- node.3: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- op.3.0.0.Unnest: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- IteratorMetrics.1.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- RowToColumnar.1.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- VeloxBatchResizer.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- IteratorMetrics.1: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- NativePlanEvaluator-1.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 2.4 MiB
+- UniffleShuffleWriter.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- RowToColumnar.1: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- UniffleShuffleWriter.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- default: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- VeloxBatchResizer.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B

at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:104)
at org.apache.gluten.memory.arrow.alloc.ManagedAllocationListener.onPreAllocation(ManagedAllocationListener.java:61)
at org.apache.gluten.shaded.org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:300)
at org.apache.gluten.shaded.org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)
at org.apache.gluten.shaded.org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:280)
at org.apache.gluten.shaded.org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)
at org.apache.gluten.execution.RowToVeloxColumnarExec$$anon$1.next(RowToVeloxColumnarExec.scala:190)
at org.apache.gluten.execution.RowToVeloxColumnarExec$$anon$1.next(RowToVeloxColumnarExec.scala:139)
at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.next(IteratorsV1.scala:173)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.next(IteratorsV1.scala:74)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.next(IteratorsV1.scala:42)
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
at org.apache.gluten.vectorized.ColumnarBatchInIterator.next(ColumnarBatchInIterator.java:41)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:154)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:66)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:38)
at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:95)
at org.apache.gluten.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:122)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at org.apache.gluten.vectorized.ColumnarBatchInIterator.hasNext(ColumnarBatchInIterator.java:36)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:154)
at org.apache.gluten.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:122)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:38)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:66)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.writer.VeloxUniffleColumnarShuffleWriter.writeImpl(VeloxUniffleColumnarShuffleWriter.java:145)
at org.apache.spark.shuffle.writer.RssShuffleWriter.write(RssShuffleWriter.java:344)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```

### Gluten version

Gluten-1.5

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

RowToVeloxColumnarExec.scala is the only project file named, with the failure at lines 139 and 190; start by tracing the allocation through ThrowOnOomMemoryTarget and ManagedAllocationListener in the stack trace. Reproduce with the reported Gluten 1.5 and Spark 3.5.x versions, collecting the missing configuration and system details. Done means the RowToVeloxColumnar path no longer fails with the reported off-heap error under a documented reproducer.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.