Regenerate input partitions for small files
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Currently when vanilla spark partition the parquet files, it sort the input splits firstly, then adjacent splits are coalesced into a single partition. If the input splits size distribution is uneven, some partition will have few splits while some partitions has lots of splits. It cause more serious issue in Gluten than Vanilla Spark. Below chart shows the impact clearly, partition elapsed time and partition numbers are directly proportional.
The new partitioning strategy should take both partition size and file count into account.
### Gluten version
None
Contributor guide
Assessment
This issue has not been assessed yet.