apache / apache/datafusion

Parquet Statistics Pruning Ignores ColumnOrder, resulting in potentially incorrect statistics

Aperta
#8,342 5 commenti 1 reazione 1 assegnatario Rivendicata da @kumarUjjawal Vedi su GitHub
bug
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

### Describe the bug

The statistics are only valid if interpreted in the context of [ColumnOrder](https://docs.rs/parquet/latest/parquet/basic/enum.ColumnOrder.html), otherwise the results are not necessarily correct

The `ColumnOrder` field in Parquet statistics says what ordering was used to compute the min/max values and seems not to be widely used or populated in the eco system. However, ignoring it when it is present is probably wrong

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.