apache / apache/gluten

[VL][BUG] task crash when use Dynamically sizing off-heap memory

Open
#11,445 3 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

task crash when use Dynamically sizing off-heap memory, The target buffer size is insufficient: 0 vs.2981

### Gluten version

main branch

### Spark version

Spark-3.4.x

### Spark configurations

${SPARK_HOME}/bin/spark-shell \
--master yarn --deploy-mode client \
--queue root.test1 \
--master yarn \
--driver-memory 8G \
--executor-cores 4 \
--num-executors 100 \
--conf spark.executor.memory=20g \
--conf spark.executor.memoryOverhead=6g \
--conf spark.memory.offHeap.enabled=true \
--conf spark.sql.catalogImplementation="in-memory" \
--conf spark.driver.maxResultSize=2g \
--conf spark.sql.shuffle.partitions=400 \
--conf spark.dynamicAllocation.enabled=false \
--conf spark.executorEnv.JAVA_HOME=/software/servers/jdk-17.0.12 \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.gluten.memory.dynamic.offHeap.sizing.enabled=true \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g

### System information

centos7
jdk17

### Relevant logs

26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO UnifiedMemoryManager: Will not store test_964a8384-26c9-4b91-9ef2-4ce0c93516a9 as the required space (8388608 bytes) exceeds our memory limit (0 bytes)
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] ERROR GlobalOffHeapMemoryTarget: Spark off-heap memory is exhausted. Storage: 0 / 0, execution: 0 / 0
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] WARN ThrowOnOomMemoryTarget: Max number of sleeps 9 has reached.
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO TaskMemoryManager: Memory used in task 12688
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO TaskMemoryManager: Acquired by org.apache.gluten.memory.memtarget.spark.TreeMemoryConsumer@45296cac: 16.0 MiB
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO TaskMemoryManager: 0 bytes of memory were used by task 12688 but are not associated with specific consumers
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO TaskMemoryManager: 16777216 bytes of memory are used for execution and 4658137 bytes of memory are used for storage
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] ERROR TaskResources: Task 12688 failed by error:
org.apache.gluten.exception.GlutenException: The target buffer size is insufficient: 0 vs.2981
at org.apache.gluten.vectorized.ColumnarBatchSerializerJniWrapper.serialize(Native Method)
at org.apache.spark.sql.execution.BroadcastUtils$.$anonfun$serializeStream$3(BroadcastUtils.scala:171)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
at scala.collection.TraversableOnce.to(TraversableOnce.scala:366)
at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364)
at scala.collection.AbstractIterator.to(Iterator.scala:1431)
at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
at org.apache.spark.sql.execution.BroadcastUtils$.serializeStream(BroadcastUtils.scala:176)
at org.apache.gluten.backendsapi.velox.VeloxSparkPlanExecApi.$anonfun$createBroadcastRelation$1(VeloxSparkPlanExecApi.scala:677)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:862)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:862)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:140)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:562)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1555)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:565)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:842)

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with the listed Spark configurations and inspect the stack path through BroadcastUtils.scala, ColumnarBatchSerializerJniWrapper, and VeloxSparkPlanExecApi.scala. Trace why dynamically sized off-heap memory produces a zero target buffer during broadcast serialization; done means the task no longer crashes under this configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.