apache / apache/datafusion

Regression: DataFusion 55 no longer prunes row groups for `col = <literal>` when statistics show the column is entirely NULL

Closed
#24,769 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

> [!NOTE]
> This report was investigated and written with AI assistance (Claude Code), posted with the account owner's review and consent.

## Describe the bug

DataFusion 54 prunes a Parquet row group from statistics alone when the predicate is `col = ` and the row group's statistics record `null_count == row_count` for that column (equality with a non-NULL literal cannot match any row). DataFusion 55.0.0 builds the **same pruning predicate** — including the `_null_count@N != row_count@M` clause — but no longer prunes the row group; it is scanned instead.

Query **results are unaffected** (0 rows on both versions). The regression is scan work: for workloads where a selective equality column is sparsely populated (in our case, a `body` column that is NULL for the overwhelming majority of rows), row groups that 54 skipped from the footer are now read.

## To Reproduce

Self-contained reproducer (~70 lines, inlined below): writes a single-row-group Parquet file with one nullable `Binary` column, all 100 values NULL, **default `WriterProperties` and default `SessionContext`**, registers it as a `ListingTable`, and filters with the DataFrame API:

```rust
df.filter(col("body").eq(lit(ScalarValue::Binary(Some(b"x".to_vec())))))?
```

Output, `datafusion = "54"`:

```
rows = 0 (correct on both versions)
PruningMetrics { name: "row_groups_pruned_statistics", pruning_metrics: PruningMetrics { pruned: 1, matched: 0, fully_matched: 0 } }
```

Output, `datafusion = "55"` (only the dependency line changed):

```
rows = 0 (correct on both versions)
PruningMetrics { name: "row_groups_pruned_statistics", pruning_metrics: PruningMetrics { pruned: 0, matched: 1, fully_matched: 0 } }
```

Cargo.toml + src/main.rs (complete)

```toml
[package]
name = "df-pruning-repro"
version = "0.0.0"
edition = "2021"

[dependencies]
# Flip to "54" and the row group is pruned; on "55" it is scanned.
datafusion = "55"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
tempfile = "3"
```

```rust
//! DataFusion 54 prunes a row group whose statistics say a column is
//! entirely NULL when the predicate is `col = `; DataFusion 55
//! scans it. Default writer properties, default SessionContext.
use std::sync::Arc;

use datafusion::arrow::array::{BinaryArray, RecordBatch};
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::common::ScalarValue;
use datafusion::datasource::file_format::parquet::ParquetFormat;
use datafusion::datasource::listing::{
ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl,
};
use datafusion::parquet::arrow::ArrowWriter;
use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
// One row group, one nullable Binary column, every value NULL —
// the footer statistics record null_count == num_rows, no min/max.
let schema = Arc::new(Schema::new(vec![Field::new(
"body",
DataType::Binary,
true,
)]));
let batch = RecordBatch::try_new(
schema.clone(),
vec![Arc::new(BinaryArray::from(vec![None::<&[u8]>; 100]))],
)
.unwrap();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("all_null.parquet");
let file = std::fs::File::create(&path).unwrap();
let mut w = ArrowWriter::try_new(file, schema, None).unwrap();
w.write(&batch).unwrap();
w.close().unwrap();

let ctx = SessionContext::new();
let url = ListingTableUrl::parse(format!("file://{}", path.display())).unwrap();
let options =
ListingOptions::new(Arc::new(ParquetFormat::default())).with_file_extension(".parquet");
let schema = options.infer_schema(&ctx.state(), &url).await?;
let table = ListingTable::try_new(
ListingTableConfig::new(url)
.with_listing_options(options)
.with_schema(schema),
)?;
// `body = X` can match nothing when every value is NULL, so the row
// group is prunable from statistics alone.
let df = ctx
.read_table(Arc::new(table))?
.filter(col("body").eq(lit(ScalarValue::Binary(Some(b"x".to_vec())))))?;

let plan = df.create_physical_plan().await?;
let batches = datafusion::physical_plan::collect(plan.clone(), ctx.task_ctx()).await?;
println!(
"rows = {} (correct on both versions)",
batches.iter().map(|b| b.num_rows()).sum::()
);

fn walk(p: &Arc) {
if let Some(m) = p.metrics() {
for metric in m.iter() {
if metric.value().name() == "row_groups_pruned_statistics" {
println!("{:?}", metric.value());
}
}
}
for c in p.children() {
walk(c);
}
}
walk(&plan);
Ok(())
}
```

## Expected behavior

The row group is pruned on 55 as it was on 54: the file's statistics prove `body = 'x'` cannot match (`null_count == row_count`), and the physical plan's `pruning_predicate` (identical on both versions in our larger application, including the `body_null_count != row_count` conjunct) already expresses that proof.

## Additional context

- Found upgrading a Parquet log store (ourios) from DF 54.0.0 → 55.0.0: three pruning-assertion tests went red with `pruned: 0` where 54 gave `pruned: 2`; written files byte-identical across the upgrade (parquet 58 vs 59 emit the same statistics for these columns), so this is read-path only.
- Triage hint: the regression reproduces through **`ListingTable` + the DataFrame API filter**. In our first attempt we could NOT reproduce via `register_parquet` + a SQL string (`WHERE body = X'6E6F7065'`) — that path does not prune on **either** 54 or 55 — so the SQL literal/rewrite path seems to sit on a different guarantee/pruning route and may mask the regression during triage.
- `EnabledStatistics::Page` vs `Chunk` and dictionary on/off for the column make no difference; defaults reproduce.

Contributor guide

Open the contributing guide

Research direction

Start with the inline Cargo.toml and src/main.rs reproducer, running it against DataFusion 54 and 55 through ListingTable and the DataFrame filter. Trace the pruning_predicate and statistics row-group pruning path, including EnabledStatistics, and compare the reported row_groups_pruned_statistics metric. Done means the 55 path prunes the all-NULL row group as 54 does.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
63/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.