apache / apache/datafusion

Support parquet content-defined chunking options

Open
#21,408 1 comment 0 reactions 1 assignee Claimed by @shruti2522 View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

This issue is filed to track the changes proposed in https://github.com/apache/datafusion/pull/21110.

DataFusion currently does not expose the new Parquet Content-Defined Chunking (CDC) support added in `parquet-rs` by https://github.com/apache/arrow-rs/pull/9450. Traditional Parquet writing splits data pages at fixed sizes, so inserting or deleting a row causes subsequent pages to shift and can force nearly all bytes to be re-uploaded in content-addressable storage systems.

CDC instead determines page boundaries using a rolling hash over column values, so unchanged data can produce identical pages across writes. This can reduce storage and upload costs and improve deduplication behavior for rewritten datasets.

### Describe the solution you'd like

Expose the Parquet CDC writer options in DataFusion so users can enable the feature when writing Parquet files.

This should cover the configuration surface introduced upstream in `parquet-rs`, including:

- enabling content-defined chunking with the default settings
- configuring explicit CDC parameters such as `min_chunk_size`, `max_chunk_size`, and `norm_level`

The implementation and rationale are largely derived from https://github.com/apache/arrow-rs/pull/9450, and this issue exists to track carrying those changes through in DataFusion via https://github.com/apache/datafusion/pull/21110.

### Describe alternatives you've considered

Continue using the existing fixed-size Parquet page splitting behavior and do not expose CDC-related writer options in DataFusion.

That preserves current behavior, but it means users cannot take advantage of the improved page stability and deduplication characteristics now available in `parquet-rs`.

### Additional context

- Tracking PR in DataFusion: https://github.com/apache/datafusion/pull/21110
- Upstream implementation in arrow-rs: https://github.com/apache/arrow-rs/pull/9450
- Related C++ implementation referenced by the upstream PR: https://github.com/apache/arrow/pull/45360
- Background on the feature: https://huggingface.co/blog/parquet-cdc

Most of the technical content above is intentionally derived from https://github.com/apache/arrow-rs/pull/9450, with the additional context that this issue tracks the corresponding DataFusion work in https://github.com/apache/datafusion/pull/21110.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.