apache / apache/datafusion-comet
`native_datafusion` doesn't use all available parallelism for scan
Open
enhancement
native_datafusion
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Observed the issue when Comet is not fully utilizing Spark cluster parallelism.
Input: 1200 HDFS files, number of Spark planned tasks: 1800. Every file is splittable, so Spark utilizes all 1800 scanning and writing the shuffle whereas Comet utilizing only 1200 tasks having 600 idle.
I was not able to reproduce the same locally, will try on local HDFS later
### Describe the potential solution
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.