apache / apache/gluten

[VL] Container killed by YARN for exceeding physical memory limits

Open
#10,010 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

When I run the SQL like below, an OOM error occurs.
```sql
select 20250530 ,key_id2, count(distinct key_id3)
from
(
select *
from xxxtable1
where dt between '20180101' and '20250530'
) a
left join
(
select *
from xxxtable2
where ds between '20180101' and '20250530'
) b on lower(a.key_id1)=lower(b.key_id1) and a.key_id2=b.key_id2
group by key_id2
```
Error msg is :
`ExecutorLostFailure (executor 43 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding physical memory limits. 6.0 GB of 6 GB physical memory used. Consider boosting spark.executor.memoryOverhead.`

The data size of scan in these two tables is about 1 TB. I tried to use shufflehashjoin and sortmergerjoin respectively, but they were failed. The same SQL can be run successfully in vanilla spark.

The failed stage is join, I suspect it has something wrong with spill.

Image

### Gluten version

Gluten-1.3

### Spark version

Spark-3.2.x

### Spark configurations

spark.memory.offHeap.enabled=true;
spark.memory.offHeap.size=3g;
spark.yarn.executor.memoryOverhead=2g;
spark.executor.memory=1g;
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager;
spark.sql.shuffle.partitions=500;
spark.io.compression.codec=zstd;

### System information

Gluten Version: 1.3.0
Commit: 98546a6d62e889d792d44715d90b1bf92f2e74e3
CMake Version: 3.28.3
System: Linux-4.9.0-14-amd64
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 11.5.0
C Compiler: /usr/bin/cc
C Compiler Version: 11.5.0
CMake Prefix Path: /usr/local;/usr;/;/usr/local;/usr/local;/usr/X11R6;/usr/pkg;/opt

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Begin with the supplied SQL reproduction and the failed join stage, then inspect how spill behavior interacts with the listed Spark, Gluten, and executor memory settings. Compare the join execution with vanilla Spark; done means the physical-memory failure is explained and the query completes successfully under the relevant configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.