[C++][Parquet] Page statistics not written in v2?
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
While working in integration tests of parquet2 against pyarrow, I noticed that page statistics are only written by pyarrow when using version 1.
I do not have an easy way to reproduce this within pyarrow as I am not sure how to access individual pages from a column chunk, but it is something that I observe when trying to integrate.
The row group stats are still written, this only affects page statistics.
pyarrow call:
```
pa.parquet.write_table(
t,
path,
version="2.0",
data_page_version="2.0",
write_statistics=True,
)
```
changing version to "1.0" does not impact this behavior, suggesting that the specific option causing this behavior is the data_page_version="2.0".
**Reporter**: [Jorge Leitão](https://issues.apache.org/jira/browse/ARROW-13240) / @jorgecarleitao
**Note**: *This issue was originally created as [ARROW-13240](https://issues.apache.org/jira/browse/ARROW-13240). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start from the pyarrow.parquet.write_table call in the report and reproduce the difference between data_page_version="1.0" and "2.0" with write_statistics enabled. Compare page statistics with the row-group statistics, and consider the issue complete when v2 data pages preserve page statistics as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100