Parquet Writer: Content-defined Chunking (CDC)
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Add a way to automatically chunk parquet data based on content to allow the usage of content-addressable storage (CAS). This shall reduce upload throughput for subsequent, partial changes of parquet files (e.g. just updating a few rows).
This is basically https://github.com/apache/arrow/issues/45750 but for arrow-rs.
**Describe the solution you'd like**
Add some kind of wrapper or parameter to the parquet writer that automatically chunks the data.
**Describe alternatives you've considered**
\-
**Additional context**
- https://huggingface.co/blog/parquet-cdc
- https://github.com/apache/arrow/pull/45360
Contributor guide
Assessment
This issue has not been assessed yet.