apache / apache/arrow

[C++][Parquet] Process parquet rowgroups without Arrow conversion

Open
#35,638 27 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Component: Python Status: needs champion Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the usage question you have. Please include as many useful details as possible.

I'd like to read a Parquet file and append an Arrow table to the new Parquet file created based on the old file and the new table added as a new row group.
Can I read the Parquet rowgroup by rowgroup, decide to drop any or use them and assemble a new Parquet file without doing the (de)serialization to Arrow?

### Component(s)

C++, Parquet, Python

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by tracing the C++ Parquet row-group read/write path and its Arrow conversion boundary, then compare that with the requested append-and-drop workflow. Done means establishing whether the workflow is supported and documenting a supported approach, or producing a clearly scoped change request.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.