[VL][BUG] task crash when use Dynamically sizing off-heap memory
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
task crash when use Dynamically sizing off-heap memory, The target buffer size is insufficient: 0 vs.2981
### Gluten version
main branch
### Spark version
Spark-3.4.x
### Spark configurations
${SPARK_HOME}/bin/spark-shell \
--master yarn --deploy-mode client \
--queue root.test1 \
--master yarn \
--driver-memory 8G \
--executor-cores 4 \
--num-executors 100 \
--conf spark.executor.memory=20g \
--conf spark.executor.memoryOverhead=6g \
--conf spark.memory.offHeap.enabled=true \
--conf spark.sql.catalogImplementation="in-memory" \
--conf spark.driver.maxResultSize=2g \
--conf spark.sql.shuffle.partitions=400 \
--conf spark.dynamicAllocation.enabled=false \
--conf spark.executorEnv.JAVA_HOME=/software/servers/jdk-17.0.12 \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.gluten.memory.dynamic.offHeap.sizing.enabled=true \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g
### System information
centos7
jdk17
### Relevant logs
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO UnifiedMemoryManager: Will not store test_964a8384-26c9-4b91-9ef2-4ce0c93516a9 as the required space (8388608 bytes) exceeds our memory limit (0 bytes)
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] ERROR GlobalOffHeapMemoryTarget: Spark off-heap memory is exhausted. Storage: 0 / 0, execution: 0 / 0
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] WARN ThrowOnOomMemoryTarget: Max number of sleeps 9 has reached.
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO TaskMemoryManager: Memory used in task 12688
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO TaskMemoryManager: Acquired by org.apache.gluten.memory.memtarget.spark.TreeMemoryConsumer@45296cac: 16.0 MiB
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO TaskMemoryManager: 0 bytes of memory were used by task 12688 but are not associated with specific consumers
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] INFO TaskMemoryManager: 16777216 bytes of memory are used for execution and 4658137 bytes of memory are used for storage
26/01/19 22:11:28 [Executor task launch worker for task 0.2 in stage 8.0 (TID 12688)] ERROR TaskResources: Task 12688 failed by error:
org.apache.gluten.exception.GlutenException: The target buffer size is insufficient: 0 vs.2981
at org.apache.gluten.vectorized.ColumnarBatchSerializerJniWrapper.serialize(Native Method)
at org.apache.spark.sql.execution.BroadcastUtils$.$anonfun$serializeStream$3(BroadcastUtils.scala:171)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
at scala.collection.TraversableOnce.to(TraversableOnce.scala:366)
at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364)
at scala.collection.AbstractIterator.to(Iterator.scala:1431)
at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
at org.apache.spark.sql.execution.BroadcastUtils$.serializeStream(BroadcastUtils.scala:176)
at org.apache.gluten.backendsapi.velox.VeloxSparkPlanExecApi.$anonfun$createBroadcastRelation$1(VeloxSparkPlanExecApi.scala:677)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:862)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:862)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:140)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:562)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1555)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:565)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:842)
```bash
```
Contributor guide
Research direction
Start by reproducing the failure with the listed Spark configurations and inspect the stack path through BroadcastUtils.scala, ColumnarBatchSerializerJniWrapper, and VeloxSparkPlanExecApi.scala. Trace why dynamically sized off-heap memory produces a zero target buffer during broadcast serialization; done means the task no longer crashes under this configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100