[VL] spark not picking provided shuffle partitions
Open
bug
triage
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
[Expected behavior]
I am trying to run a job on both plain vannila spark and with gluten. There was one specific shuffle stage where in the plain spark it got completed in less than a minute but in gluten it took 20 mins. The number of tasks created was 2000(which is my spark.sql.shuffle.partitions) whereas in gluten it was 6. What could be the reason for this?
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Assessment
This issue has not been assessed yet.