apache / apache/arrow-rs

Parquet Writer: Content-defined Chunking (CDC)

Open
#8,010 0 comments 2 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Add a way to automatically chunk parquet data based on content to allow the usage of content-addressable storage (CAS). This shall reduce upload throughput for subsequent, partial changes of parquet files (e.g. just updating a few rows).

This is basically https://github.com/apache/arrow/issues/45750 but for arrow-rs.

**Describe the solution you'd like**
Add some kind of wrapper or parameter to the parquet writer that automatically chunks the data.

**Describe alternatives you've considered**
\-

**Additional context**
- https://huggingface.co/blog/parquet-cdc
- https://github.com/apache/arrow/pull/45360

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.