apache / apache/auron

native engine panics Execution overflow

Open
#1,893 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

1. Environment
6 worker node YARN cluster, x86 architecture, each node with 64 cores and 500GB memory.
Hadoop 3.2.2
Spark 3.5.4
Blaze 5.0.0

2. how to reproduce
Running TPC-DS benchmark, 10TB dataset, Parquet + ZSTD compression.
```
spark.executor.cores=1
spark.executor.memory=16g
spark.executor.memoryOverhead=16g
spark.driver.cores=1
spark.driver.memory=20g
spark.blaze.enable true
spark.sql.extensions org.apache.spark.sql.blaze.BlazeSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleManager
spark.memory.offHeap.enabled false
```
Queries q24a and q24b failed.
The error message is as shown in the figure.
The failure can be reproduced every time. The failed stage has 200 tasks, with 164 succeeded and 36 failed.

3. other scenario where the queries succeed
On 10TB dataset, without Blaze enabled, the queries succeed.
On the 1TB dataset, with Blaze enabled, the queries also succeed .

![Image](https://github.com/user-attachments/assets/f13d24e5-fa26-4156-978b-a4506b488dc5)
![Image](https://github.com/user-attachments/assets/eaa85ef8-6cb0-4c27-911b-41b97c35661c)

![Image](https://github.com/user-attachments/assets/71779fc3-a6c2-4a6d-abe1-19ea4ac2be25)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.