apache / apache/spark

Is Spark limited to split the Parquet read granularity by Row Group level only?

Open
#55,747 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
44k
Forks
29.4k
PR merge metrics
No merged PRs in 30d

Description

According to some articles I found:

- https://cloudsqale.com/2021/03/19/spark-reading-parquet-why-the-number-of-tasks-can-be-much-larger-than-the-number-of-row-groups/
- https://www.gresearch.com/news/parquet-files-know-your-scaling-limits/

Seems Spark can only parallelize the reads across row groups. Is this a known limitation?

Is there any way to split it by row or page level?

If a file has a single row group, it means all the tasks, except 1, would be idle? And that one task would read the entire file?

Thanks.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing Spark's Parquet scan partitioning behavior and compare row-group, page, and row-level granularity. Done means documenting whether finer-grained splitting is supported and what happens with a single-row-group file.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.