[C++][Parquet] Process parquet rowgroups without Arrow conversion
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the usage question you have. Please include as many useful details as possible.
I'd like to read a Parquet file and append an Arrow table to the new Parquet file created based on the old file and the new table added as a new row group.
Can I read the Parquet rowgroup by rowgroup, decide to drop any or use them and assemble a new Parquet file without doing the (de)serialization to Arrow?
### Component(s)
C++, Parquet, Python
Contributor guide
Research direction
The issue names no files or tests. Start by tracing the C++ Parquet row-group read/write path and its Arrow conversion boundary, then compare that with the requested append-and-drop workflow. Done means establishing whether the workflow is supported and documenting a supported approach, or producing a clearly scoped change request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100