Parquet runtime pruning can drop NULLS FIRST TopK rows when null_count is missing
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Parquet runtime row-group pruning treats an absent `null_count` as zero. This can incorrectly discard nulls needed by a TopK with `NULLS FIRST`.
The assumption is present in [DataFusion 55.1.0's `RowGroupPruner::should_prune`](https://github.com/apache/datafusion/blob/55.1.0/datafusion/datasource-parquet/src/push_decoder.rs#L212-L225), and remains on [current main](https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/push_decoder.rs#L247-L265). It sets `missing_null_counts_as_zero: true`, which the [statistics adapter](https://github.com/apache/datafusion/blob/55.1.0/datafusion/datasource-parquet/src/row_group_filter.rs#L477-L509) passes to Parquet's `StatisticsConverter`.
The [Parquet specification](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L283-L291) requires an absent null count to remain distinct from a known zero.
### Concrete case to reproduce
Use one file and one scan partition, with a nullable INT column `k` and these row groups in scan order:
| Row group | Values | Column-chunk statistics |
| --- | --- | --- |
| 0 | `0, 1, 2` | Ordinary statistics |
| 1 | `NULL, 100, 101` | `min=100`, `max=101`, but `null_count` absent |
| 2 | `200, 201, 202` | Ordinary statistics |
Keep min/max statistics on group 1 and remove only its optional null-count field. Disabling all statistics is a different case.
Run:
```sql
SELECT k FROM t ORDER BY k ASC NULLS FIRST LIMIT 1;
```
Connect TopK's live predicate to the Parquet scan, use small batches, and disable page-index and row-filter pushdown to isolate runtime row-group pruning.
After the first group, TopK retains `0` and [publishes](https://github.com/apache/datafusion/blob/55.1.0/datafusion/physical-plan/src/topk/mod.rs#L605-L635):
```sql
k IS NULL OR k < 0
```
For group 1, treating the unknown null count as zero makes `IS NULL` appear impossible, while `min=100` rules out `k < 0`. The group can therefore be pruned even though its null must win. The expected result is `NULL`; source tracing predicts `0` if that group is skipped.
### Expected behavior and fix direction
Preserve unknown null counts during pruning. A group must remain readable when its statistics cannot rule out matching nulls. Review the other callers that enable the same assumption, including static statistics pruning.
Add a regression with min/max present and null count omitted, covering TopK filtering off/on and both ascending and descending `NULLS FIRST`. Keep a known-zero-null-count control to verify that valid pruning remains effective.
### Validation and downstream context
This is a source-derived report against DataFusion 55.1.0 and the current main implementation. An isolated native pruning probe compiled, but its process was killed before producing output. I have not completed an end-to-end reproduction, so the predicted wrong result above is not a measured test result.
Found while reviewing [apache/datafusion-comet#5785](https://github.com/apache/datafusion-comet/pull/5785), which connects Spark's local TopK to this reader path.
Contributor guide
Assessment
This issue has not been assessed yet.