Min row group size parquet
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In delta-rs python bindings we switched from pyarrow writing parquets to parquet crate.
Quite often users used the min_row_goup_size functionality of the pyarrow parquet writer but the Parquet crate doesn't have this.
Any chance this can be added?
**Describe the solution you'd like**
Add min_row_group_size to parquet Writer properties that will try to distribute all the rows of a recordbatch in the range of min and max row group size
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start by locating the parquet Writer properties and the Python binding path that constructs them. Trace how record batches become row groups, then verify that the new setting distributes rows within the requested minimum and maximum sizes, with focused tests covering the resulting row-group sizes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100