apache / apache/arrow

[C++][Parquet] Page statistics not written in v2?

Open
#28,925 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

While working in integration tests of parquet2 against pyarrow, I noticed that page statistics are only written by pyarrow when using version 1.

I do not have an easy way to reproduce this within pyarrow as I am not sure how to access individual pages from a column chunk, but it is something that I observe when trying to integrate.

The row group stats are still written, this only affects page statistics.

pyarrow call:

```
pa.parquet.write_table(
t,
path,
version="2.0",
data_page_version="2.0",
write_statistics=True,
)
```

changing version to "1.0" does not impact this behavior, suggesting that the specific option causing this behavior is the data_page_version="2.0".

**Reporter**: [Jorge Leitão](https://issues.apache.org/jira/browse/ARROW-13240) / @jorgecarleitao

**Note**: *This issue was originally created as [ARROW-13240](https://issues.apache.org/jira/browse/ARROW-13240). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start from the pyarrow.parquet.write_table call in the report and reproduce the difference between data_page_version="1.0" and "2.0" with write_statistics enabled. Compare page statistics with the row-group statistics, and consider the issue complete when v2 data pages preserve page statistics as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.