apache / apache/gluten

[VL] Dynamic Offheap Sizing Feature Returns False OOM When SQL Conf is Empty

Open
#9,065 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

```
data = [{"col1": 1}, {"col1": 2}, {"col1": 3}, {"col1": 4}]
df=spark.createDataFrame(data)
df.select("col1").distinct().rdd.flatMap(lambda x:x).collect()
```

Returns
```
2025-03-12 10:45:37,789 ERROR ManagedReservationListener [Executor task launch worker for task 6.0 in stage 3.0 (TID 15)]: Error reserving memory from target
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
  spark.gluten.memory.offHeap.size.in.bytes=0.0 B
  spark.memory.offHeap.size=0.0 B
  spark.gluten.memory.task.offHeap.size.in.bytes=0.0 B
  spark.gluten.memory.conservative.task.offHeap.size.in.bytes=0.0 B
  spark.memory.offHeap.enabled=false
  spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
  Task.15: Current used bytes: 0.0 B, peak bytes: N/A
  \- Gluten.Tree.2: Current used bytes: 0.0 B, peak bytes: 0.0 B
   \- root.2: Current used bytes: 0.0 B, peak bytes: 0.0 B
   +- ShuffleWriter.2: Current used bytes: 0.0 B, peak bytes: 0.0 B
   \- OverAcquire.DummyTarget.5: Current used bytes: 0.0 B, peak bytes: 0.0 B

  at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:114)
  at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:43)
  at org.apache.gluten.exec.RuntimeJniWrapper.createRuntime(Native Method)
```
because SQLConf does not have these configs. It is better to get these from `SparkEnv`.

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by tracing how SQLConf and SparkEnv provide off-heap settings to RuntimeJniWrapper.createRuntime, then reproduce the provided Spark 3.5.x distinct/RDD example with the SQL configuration empty. Done means the empty SQLConf case obtains the correct settings instead of granting 0 bytes and raising a false OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.