apache / apache/gluten

[VL] spark not picking provided shuffle partitions

Open
#10,068 3 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

[Expected behavior]
I am trying to run a job on both plain vannila spark and with gluten. There was one specific shuffle stage where in the plain spark it got completed in less than a minute but in gluten it took 20 mins. The number of tasks created was 2000(which is my spark.sql.shuffle.partitions) whereas in gluten it was 6. What could be the reason for this?

### Gluten version

main branch

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.