[Python] Expose more metadata in pyarrow.parquet.ParquetFile.metadata
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the enhancement requested
I'm not sure if this issue pertains to all implementations of arrow including pyarrow or just c++ but related to this https://github.com/apache/arrow/issues/14870
I'm guessing it affects pyarrow as `pq.ParquetFile.metadata.to_dict()['row_groups'][0]['columns'][0]['statistics'].keys()`
doesn't have the min_value, max_value keys.
So the feature request is to include min_value and max_value in that metadata.
Additionally, I think there's metadata on whether or not a column is sorted (I might be confused on that point) but if there is it'd be good to see that too.
### Component(s)
Python
Contributor guide
Research direction
Start with pyarrow.parquet.ParquetFile.metadata and inspect the to_dict() representation described in the issue, alongside the related Apache Arrow issue 14870. Determine whether min_value and max_value are available from the underlying metadata and whether sorted-column metadata exists; done means the requested metadata is exposed for Python users with coverage for the resulting representation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100