apache / apache/arrow

[Python] Expose more metadata in pyarrow.parquet.ParquetFile.metadata

Open
#34,180 6 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

I'm not sure if this issue pertains to all implementations of arrow including pyarrow or just c++ but related to this https://github.com/apache/arrow/issues/14870

I'm guessing it affects pyarrow as `pq.ParquetFile.metadata.to_dict()['row_groups'][0]['columns'][0]['statistics'].keys()`
doesn't have the min_value, max_value keys.

So the feature request is to include min_value and max_value in that metadata.

Additionally, I think there's metadata on whether or not a column is sorted (I might be confused on that point) but if there is it'd be good to see that too.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with pyarrow.parquet.ParquetFile.metadata and inspect the to_dict() representation described in the issue, alongside the related Apache Arrow issue 14870. Determine whether min_value and max_value are available from the underlying metadata and whether sorted-column metadata exists; done means the requested metadata is exposed for Python users with coverage for the resulting representation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.