apache / apache/parquet-format

Document the behavior of Nested ColumnChunk statistics

Open
#476 3 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

### Describe the enhancement requested

At the moment, it is not clear to me what the semantics are of the `ColumnChunk`-level statistics of nested columns.

It appears that it should be based on the leaf column (which makes sense to me), but then the `null_count` (and `distinct_count` probably) are seemingly based partially on the nested level.

```python
import polars as pl
import io
import pyarrow.parquet as pq

df = pl.DataFrame([
pl.Series('a', [[1, 2, 3], None], pl.Array(pl.Int32, 3)),
])

f = io.BytesIO()
pq.write_table(df.to_arrow(), f)

f.seek(0)
pq.read_metadata(f).row_group(0).column(0).statistics
```

```console

has_min_max: True
min: 1
max: 3
null_count: 1
distinct_count: None
num_values: 3
physical_type: INT32
logical_type: None
converted_type (legacy): NONE
```

I would expect the `null_count` to equal `3` here if it was based on the leaf column.

Because of this, `null_count` basically has to be ignored for nested columns. Ideally, we would have a list of `null_count` for every nullable level of the nesting, but otherwise just specifying the semantics is good.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing specification sections for ColumnChunk statistics and nested columns, then reproduce the shown Python example with Polars and PyArrow. Trace how null_count, distinct_count, and num_values are described for leaf and nested levels. Done means the specification clearly states these semantics or documents the limitation and expected interpretation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.