[VL] Container killed by YARN for exceeding physical memory limits
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
When I run the SQL like below, an OOM error occurs.
```sql
select 20250530 ,key_id2, count(distinct key_id3)
from
(
select *
from xxxtable1
where dt between '20180101' and '20250530'
) a
left join
(
select *
from xxxtable2
where ds between '20180101' and '20250530'
) b on lower(a.key_id1)=lower(b.key_id1) and a.key_id2=b.key_id2
group by key_id2
```
Error msg is :
`ExecutorLostFailure (executor 43 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding physical memory limits. 6.0 GB of 6 GB physical memory used. Consider boosting spark.executor.memoryOverhead.`
The data size of scan in these two tables is about 1 TB. I tried to use shufflehashjoin and sortmergerjoin respectively, but they were failed. The same SQL can be run successfully in vanilla spark.
The failed stage is join, I suspect it has something wrong with spill.
### Gluten version
Gluten-1.3
### Spark version
Spark-3.2.x
### Spark configurations
spark.memory.offHeap.enabled=true;
spark.memory.offHeap.size=3g;
spark.yarn.executor.memoryOverhead=2g;
spark.executor.memory=1g;
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager;
spark.sql.shuffle.partitions=500;
spark.io.compression.codec=zstd;
### System information
Gluten Version: 1.3.0
Commit: 98546a6d62e889d792d44715d90b1bf92f2e74e3
CMake Version: 3.28.3
System: Linux-4.9.0-14-amd64
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 11.5.0
C Compiler: /usr/bin/cc
C Compiler Version: 11.5.0
CMake Prefix Path: /usr/local;/usr;/;/usr/local;/usr/local;/usr/X11R6;/usr/pkg;/opt
### Relevant logs
```bash
```
Contributor guide
Research direction
Begin with the supplied SQL reproduction and the failed join stage, then inspect how spill behavior interacts with the listed Spark, Gluten, and executor memory settings. Compare the join execution with vanilla Spark; done means the physical-memory failure is explained and the query completes successfully under the relevant configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100