apache / apache/datafusion

Parquet Statistics Pruning Ignores ColumnOrder, resulting in potentially incorrect statistics

Open
#8,342 5 comments 1 reaction 1 assignee Claimed by @kumarUjjawal View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

The statistics are only valid if interpreted in the context of [ColumnOrder](https://docs.rs/parquet/latest/parquet/basic/enum.ColumnOrder.html), otherwise the results are not necessarily correct

The `ColumnOrder` field in Parquet statistics says what ordering was used to compute the min/max values and seems not to be widely used or populated in the eco system. However, ignoring it when it is present is probably wrong

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.