Is Spark limited to split the Parquet read granularity by Row Group level only?
- Dominant language
- Scala
- Stars
- 44k
- Forks
- 29.4k
- PR merge metrics
- No merged PRs in 30d
Description
According to some articles I found:
- https://cloudsqale.com/2021/03/19/spark-reading-parquet-why-the-number-of-tasks-can-be-much-larger-than-the-number-of-row-groups/
- https://www.gresearch.com/news/parquet-files-know-your-scaling-limits/
Seems Spark can only parallelize the reads across row groups. Is this a known limitation?
Is there any way to split it by row or page level?
If a file has a single row group, it means all the tasks, except 1, would be idle? And that one task would read the entire file?
Thanks.
Contributor guide
Research direction
No files or tests are named. Start by tracing Spark's Parquet scan partitioning behavior and compare row-group, page, and row-level granularity. Done means documenting whether finer-grained splitting is supported and what happens with a single-row-group file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100