apache / apache/gluten

[VL] Executor reports an error due to OOM but task not fail

Open
#9,698 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Hello Team, I found that when the Executor reports an error due to OOM, it will only print out the error log, but will not exit the Executor, resulting in no automatic retry.
I can reproduce this problem using a small offheap memory. I am using Gluten version 1.2.1. Upgrading is not easy for me. Is this a known problem? Maybe I can cherry pick some bugfix patch?

### Gluten version

Gluten-1.2

### Spark version

Spark-3.4.x

### Spark configurations

spark.plugins=org.apache.gluten.GlutenPlugin
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=1m
spark.shuffle.manager = org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.gluten.sql.columnar.forceShuffledHashJoin=true
spark.gluten.sql.native.writer.enabled = false

### System information

Velox System Info v0.0.2
Commit: ea7315f851bc5fb5aaf7ce5a0e27a97df4a0ec31
CMake Version: 3.14.5
System: Linux-4.19.91-011.ali4000.alios7.x86_64
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
C++ Compiler: /usr/local/alicpp/built/gcc-9.2.1/gcc-9.2.1/bin/g++
C++ Compiler Version: 9.2.1
C Compiler: /usr/local/alicpp/built/gcc-9.2.1/gcc-9.2.1/bin/gcc
C Compiler Version: 9.2.1
CMake Prefix Path: /usr/local;/usr;/;/usr/local;/usr/local;/usr/X11R6;/usr/pkg;/opt

### Relevant logs

```bash
25/05/20 15:12:00 ERROR ManagedReservationListener: Error reserving memory from target
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 341.3 KiB. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=1024.0 KiB
spark.gluten.memory.task.offHeap.size.in.bytes=256.0 KiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=128.0 KiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
Task.0: Current used bytes: 0.0 B, peak bytes: N/A
\- Gluten.Tree.3: Current used bytes: 0.0 B, peak bytes: 443.7 KiB
\- root.3: Current used bytes: 0.0 B, peak bytes: 443.7 KiB
+- OverAcquire.DummyTarget.0: Current used bytes: 0.0 B, peak bytes: 102.4 KiB
\- RowToColumnar.3: Current used bytes: 0.0 B, peak bytes: 341.3 KiB

at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:43)
at org.apache.gluten.exec.RuntimeJniWrapper.createRuntime(Native Method)
at org.apache.gluten.exec.Runtime$RuntimeImpl.(Runtime.scala:63)
at org.apache.gluten.exec.Runtime$.apply(Runtime.scala:48)
at org.apache.gluten.exec.Runtimes$.create(Runtimes.scala:33)
at org.apache.gluten.exec.Runtimes$.$anonfun$contextInstance$1(Runtimes.scala:29)
at org.apache.spark.util.TaskResourceRegistry.$anonfun$addResourceIfNotRegistered$1(TaskResources.scala:320)
at org.apache.spark.util.TaskResourceRegistry.lock(TaskResources.scala:245)
at org.apache.spark.util.TaskResourceRegistry.addResourceIfNotRegistered(TaskResources.scala:316)
at org.apache.spark.util.TaskResources$.addResourceIfNotRegistered(TaskResources.scala:157)
at org.apache.gluten.exec.Runtimes$.contextInstance(Runtimes.scala:29)
at org.apache.gluten.execution.RowToVeloxColumnarExec$.toColumnarBatchIterator(RowToVeloxColumnarExec.scala:117)
at org.apache.gluten.execution.RowToVeloxColumnarExec.$anonfun$doExecuteColumnarInternal$2(RowToVeloxColumnarExec.scala:72)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:853)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:853)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.gluten.execution.ColumnarInputRDDsWrapper.$anonfun$getIterators$1(WholeStageTransformer.scala:445)
at scala.collection.immutable.List.flatMap(List.scala:366)
at org.apache.gluten.execution.ColumnarInputRDDsWrapper.getIterators(WholeStageTransformer.scala:436)
at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.$anonfun$compute$1(WholeStageZippedPartitionsRDD.scala:48)
at org.apache.gluten.utils.Arm$.withResource(Arm.scala:25)
at org.apache.gluten.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37)
at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.compute(WholeStageZippedPartitionsRDD.scala:46)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:139)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1545)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:879)
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing ThrowOnOomMemoryTarget.borrow and ManagedReservationListener.reserve in the reported stack, then follow RuntimeJniWrapper.createRuntime and Runtimes.contextInstance. Reproduce with the supplied 1 MiB off-heap configuration and inspect how the failure reaches RowToVeloxColumnarExec. Done means the OOM causes the task or executor failure path to trigger Spark's automatic retry rather than only logging the error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
backend, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.