apache / apache/gluten

[VL] One case of should spill but OOM

Open
#4,275 6 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

I suppose in below case, UnsafeExternalSorter should spill.

```
Caused by: java.lang.RuntimeException: Error during calling Java code from native code: io.glutenproject.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8388608, granted: 8126464. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application.
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=12.0 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=3.0 GiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=1536.0 MiB
Memory consumer stats:
Task.241798: Current used bytes: 3.0 GiB, peak bytes: N/A
+- org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@4479ef33: Current used bytes: 2.1 GiB, peak bytes: N/A
\- Gluten.Tree.18: Current used bytes: 952.0 MiB, peak bytes: 1048.0 MiB
\- root.18: Current used bytes: 952.0 MiB, peak bytes: 1048.0 MiB
+- ShuffleReader.18: Current used bytes: 560.0 MiB, peak bytes: 600.0 MiB
| \- single: Current used bytes: 256.0 MiB, peak bytes: 272.0 MiB
| \- ShuffleReader_default_leaf: Current used bytes: 248.1 MiB, peak bytes: 270.3 MiB
+- ColumnarToRow.18: Current used bytes: 384.0 MiB, peak bytes: 656.0 MiB
| \- single: Current used bytes: 384.0 MiB, peak bytes: 640.0 MiB
| \- ColumnarToRow_default_leaf: Current used bytes: 384.0 MiB, peak bytes: 640.0 MiB
+- ArrowContextInstance.18: Current used bytes: 8.0 MiB, peak bytes: 8.0 MiB
+- OverAcquire.DummyTarget.52: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- WholeStageIterator.18: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- WholeStageIterator_default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- task.Gluten_Stage_13_TID_241798: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- node.1: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- op.1.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- node.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- op.0.0.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- OverAcquire.DummyTarget.56: Current used bytes: 0.0 B, peak bytes: 0.0 B
\- OverAcquire.DummyTarget.55: Current used bytes: 0.0 B, peak bytes: 0.0 B

```

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing Spark's UnsafeExternalSorter interaction with Gluten's ThrowOnOomMemoryTarget and the Velox memory consumers shown in the report. Reconstruct the missing Spark configuration and workload details, then verify why the reported allocation is not followed by spilling; done means the case no longer fails with an out-of-memory error or is shown to be configuration-related.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.