apache / apache/arrow

[C++][Parquet] Provide a rewriter to rewrite parquet files without decoding all the row groups/pages

Open
#47,628 4 comments 0 reactions 1 assignee Claimed by @HuaHuaY View on GitHub
Component: C++ Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Provide a rewriter (may be named `ParquetFilesRewriter`) which allows to rewrite some parquet files to one or more new parquet files. Most of the time it copies the data directly. And it will only decrypt, decompress or decode pages when necessary. It can be used to add a column to a existed parquet file, to compact small parquet files and so on.

There is a similar implementation in Java: https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/rewrite/ParquetRewriter.java

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.