apache / apache/arrow

[Python][Docs] use_content_defined_chunking not shown correctly for ParquetWriter / write_table

Open
#47,477 0 comments 0 reactions 1 assignee Claimed by @kszucs View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Was reading about [`Content-Defined Chunking`](https://huggingface.co/blog/parquet-cdc) which sounds very interesting!

Then I came across
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html etc

`use_content_defined_chunking` was documented in the docstring, but was not part of the APIs on the top of the page.

Also, it will be useful to have a note about the consequence of setting `use_content_defined_chunking=True` on systems that do not support CDC.

(final personal thought: should this filesystem-specific option to be set somewhere else? for now this is only relevant for Hugging face now?)

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.