apache / apache/arrow

[Python][Parquet] Expose SizeStatisticsLevel in the Parquet writer API

Open
#50,619 0 comments 0 reactions 1 assignee Claimed by @abaybek View on GitHub
Component: Parquet Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

The Parquet C++ library already lets you control how much [`SizeStatistics`](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift) metadata is written, via `WriterProperties::Builder::set_size_statistics_level()` (added in #40594). The available levels are:

- `None` — write no size statistics
- `ColumnChunk` — write size statistics only at the column-chunk level (`ColumnMetaData.size_statistics`)
- `PageAndColumnChunk` — also write page-level histograms into `ColumnIndex` / `OffsetIndex` (**this is the C++ default**)

However, **PyArrow does not currently expose this setting.** There is no way from `pyarrow.parquet.write_table`, `ParquetWriter`, or the dataset writer to change the size-statistics level, so every file written by PyArrow always includes `SizeStatistics` at the `PageAndColumnChunk` level.

We have hit one concrete problem with it being unconditionally written:

**Reader compatibility.** Some third-party / older Parquet readers do not handle the newer `SizeStatistics` thrift fields gracefully and fail to read files that PyArrow produces. Being able to write with `size_statistics_level="none"` produces files those readers can consume, without having to downgrade any other writer settings.

The C++ capability exists and is stable; this request is purely to surface it through the Python binding layer.

### Component(s)

Python, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.