Pruning of floating point Parquet columns is incorrect when `NaN` is present
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
This was mentioned in https://github.com/apache/datafusion/issues/15742#issuecomment-2815595171 and discussed in detail in https://github.com/apache/parquet-format/pull/221, but datafusion is over-aggressive in pruning floating point columns. The issue appears with predicates of the form `x [gt|lt] literal`. Consider a column consisting of `[1.0, 0.0, -1.0, NaN, -2.0]`, the max will be 1 and the min -2. A query like `select * from ... where x > 2` will return no rows because no chunk exists where `max > 2`.
### To Reproduce
```sql
> select * from 'parquet-testing/data/float16_nonzeros_and_nans.parquet' where x > arrow_cast(2.0, 'Float16');
+---+
| x |
+---+
+---+
0 row(s) fetched.
```
### Expected behavior
The above query should return a single row containing `NaN`.
### Additional context
The Parquet community is considering changes to allow for `NaN` in statistics, with the currently favored approach being adding a new `ColumnOrder` to the specification. This will correct the issue above, but datafusion will need to check the `ColumnOrder` to know whether or not floating point statistics can be trusted.
Also note that if/when https://github.com/apache/parquet-format/pull/221 is merged, other predicates such as `isnan(x)` might be candidates for pruning, but that is an optimization.
Contributor guide
Research direction
Start by running the query against parquet-testing/data/float16_nonzeros_and_nans.parquet and confirm that the NaN row is incorrectly pruned. Trace floating-point statistics pruning and how Parquet ColumnOrder is handled, using the linked parquet-format discussion for context. Done means the query returns the NaN row without over-aggressive pruning, with regression coverage for the reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100