apache / apache/arrow

[C++] Configure row group size in dataset writting

Open
#29,991 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Similar to ARROW-14426, but give the ability to set a target row group size. This might not be super important to end-users, but would help us identify what sizes are good.

Alternatively in combination with ARROW-14426, being able to set a maximum row group size might give us all the tools we need.

**Reporter**: [Jonathan Keane](https://issues.apache.org/jira/browse/ARROW-14427) / @jonkeane

**Note**: *This issue was originally created as [ARROW-14427](https://issues.apache.org/jira/browse/ARROW-14427). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reading the related ARROW-14426 issue and the original ARROW-14427 report to understand the intended dataset-writing controls. Trace the dataset writing entry point from there and determine whether a target or maximum row group size is required; the work is done when the chosen size can be configured and its behavior is covered by the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.