apache / apache/datafusion

Parquet runtime pruning can drop NULLS FIRST TopK rows when null_count is missing

Closed
#25,239 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Parquet runtime row-group pruning treats an absent `null_count` as zero. This can incorrectly discard nulls needed by a TopK with `NULLS FIRST`.

The assumption is present in [DataFusion 55.1.0's `RowGroupPruner::should_prune`](https://github.com/apache/datafusion/blob/55.1.0/datafusion/datasource-parquet/src/push_decoder.rs#L212-L225), and remains on [current main](https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/push_decoder.rs#L247-L265). It sets `missing_null_counts_as_zero: true`, which the [statistics adapter](https://github.com/apache/datafusion/blob/55.1.0/datafusion/datasource-parquet/src/row_group_filter.rs#L477-L509) passes to Parquet's `StatisticsConverter`.

The [Parquet specification](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L283-L291) requires an absent null count to remain distinct from a known zero.

### Concrete case to reproduce

Use one file and one scan partition, with a nullable INT column `k` and these row groups in scan order:

| Row group | Values | Column-chunk statistics |
| --- | --- | --- |
| 0 | `0, 1, 2` | Ordinary statistics |
| 1 | `NULL, 100, 101` | `min=100`, `max=101`, but `null_count` absent |
| 2 | `200, 201, 202` | Ordinary statistics |

Keep min/max statistics on group 1 and remove only its optional null-count field. Disabling all statistics is a different case.

Run:

```sql
SELECT k FROM t ORDER BY k ASC NULLS FIRST LIMIT 1;
```

Connect TopK's live predicate to the Parquet scan, use small batches, and disable page-index and row-filter pushdown to isolate runtime row-group pruning.

After the first group, TopK retains `0` and [publishes](https://github.com/apache/datafusion/blob/55.1.0/datafusion/physical-plan/src/topk/mod.rs#L605-L635):

```sql
k IS NULL OR k < 0
```

For group 1, treating the unknown null count as zero makes `IS NULL` appear impossible, while `min=100` rules out `k < 0`. The group can therefore be pruned even though its null must win. The expected result is `NULL`; source tracing predicts `0` if that group is skipped.

### Expected behavior and fix direction

Preserve unknown null counts during pruning. A group must remain readable when its statistics cannot rule out matching nulls. Review the other callers that enable the same assumption, including static statistics pruning.

Add a regression with min/max present and null count omitted, covering TopK filtering off/on and both ascending and descending `NULLS FIRST`. Keep a known-zero-null-count control to verify that valid pruning remains effective.

### Validation and downstream context

This is a source-derived report against DataFusion 55.1.0 and the current main implementation. An isolated native pruning probe compiled, but its process was killed before producing output. I have not completed an end-to-end reproduction, so the predicted wrong result above is not a measured test result.

Found while reviewing [apache/datafusion-comet#5785](https://github.com/apache/datafusion-comet/pull/5785), which connects Spark's local TopK to this reader path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.