apache / apache/datafusion

Pruning of floating point Parquet columns is incorrect when `NaN` is present

Open
#15,812 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

This was mentioned in https://github.com/apache/datafusion/issues/15742#issuecomment-2815595171 and discussed in detail in https://github.com/apache/parquet-format/pull/221, but datafusion is over-aggressive in pruning floating point columns. The issue appears with predicates of the form `x [gt|lt] literal`. Consider a column consisting of `[1.0, 0.0, -1.0, NaN, -2.0]`, the max will be 1 and the min -2. A query like `select * from ... where x > 2` will return no rows because no chunk exists where `max > 2`.

### To Reproduce

```sql
> select * from 'parquet-testing/data/float16_nonzeros_and_nans.parquet' where x > arrow_cast(2.0, 'Float16');
+---+
| x |
+---+
+---+
0 row(s) fetched.
```

### Expected behavior

The above query should return a single row containing `NaN`.

### Additional context

The Parquet community is considering changes to allow for `NaN` in statistics, with the currently favored approach being adding a new `ColumnOrder` to the specification. This will correct the issue above, but datafusion will need to check the `ColumnOrder` to know whether or not floating point statistics can be trusted.

Also note that if/when https://github.com/apache/parquet-format/pull/221 is merged, other predicates such as `isnan(x)` might be candidates for pruning, but that is an optimization.

Contributor guide

Open the contributing guide

Research direction

Start by running the query against parquet-testing/data/float16_nonzeros_and_nans.parquet and confirm that the NaN row is incorrectly pruned. Trace floating-point statistics pruning and how Parquet ColumnOrder is handled, using the linked parquet-format discussion for context. Done means the query returns the NaN row without over-aggressive pruning, with regression coverage for the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.