apache / apache/arrow-rs

Min row group size parquet

Open
#6,292 7 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In delta-rs python bindings we switched from pyarrow writing parquets to parquet crate.

Quite often users used the min_row_goup_size functionality of the pyarrow parquet writer but the Parquet crate doesn't have this.

Any chance this can be added?

**Describe the solution you'd like**
Add min_row_group_size to parquet Writer properties that will try to distribute all the rows of a recordbatch in the range of min and max row group size

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the parquet Writer properties and the Python binding path that constructs them. Trace how record batches become row groups, then verify that the new setting distributes rows within the requested minimum and maximum sizes, with focused tests covering the resulting row-group sizes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.