apache / apache/gluten

Regenerate input partitions for small files

Open
#11,050 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Currently when vanilla spark partition the parquet files, it sort the input splits firstly, then adjacent splits are coalesced into a single partition. If the input splits size distribution is uneven, some partition will have few splits while some partitions has lots of splits. It cause more serious issue in Gluten than Vanilla Spark. Below chart shows the impact clearly, partition elapsed time and partition numbers are directly proportional.

Image

The new partitioning strategy should take both partition size and file count into account.

### Gluten version

None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.