Regression: DataFusion 55 no longer prunes row groups for `col = <literal>` when statistics show the column is entirely NULL
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
> [!NOTE]
> This report was investigated and written with AI assistance (Claude Code), posted with the account owner's review and consent.
## Describe the bug
DataFusion 54 prunes a Parquet row group from statistics alone when the predicate is `col = ` and the row group's statistics record `null_count == row_count` for that column (equality with a non-NULL literal cannot match any row). DataFusion 55.0.0 builds the **same pruning predicate** — including the `_null_count@N != row_count@M` clause — but no longer prunes the row group; it is scanned instead.
Query **results are unaffected** (0 rows on both versions). The regression is scan work: for workloads where a selective equality column is sparsely populated (in our case, a `body` column that is NULL for the overwhelming majority of rows), row groups that 54 skipped from the footer are now read.
## To Reproduce
Self-contained reproducer (~70 lines, inlined below): writes a single-row-group Parquet file with one nullable `Binary` column, all 100 values NULL, **default `WriterProperties` and default `SessionContext`**, registers it as a `ListingTable`, and filters with the DataFrame API:
```rust
df.filter(col("body").eq(lit(ScalarValue::Binary(Some(b"x".to_vec())))))?
```
Output, `datafusion = "54"`:
```
rows = 0 (correct on both versions)
PruningMetrics { name: "row_groups_pruned_statistics", pruning_metrics: PruningMetrics { pruned: 1, matched: 0, fully_matched: 0 } }
```
Output, `datafusion = "55"` (only the dependency line changed):
```
rows = 0 (correct on both versions)
PruningMetrics { name: "row_groups_pruned_statistics", pruning_metrics: PruningMetrics { pruned: 0, matched: 1, fully_matched: 0 } }
```
Cargo.toml + src/main.rs (complete)
```toml
[package]
name = "df-pruning-repro"
version = "0.0.0"
edition = "2021"
[dependencies]
# Flip to "54" and the row group is pruned; on "55" it is scanned.
datafusion = "55"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
tempfile = "3"
```
```rust
//! DataFusion 54 prunes a row group whose statistics say a column is
//! entirely NULL when the predicate is `col = `; DataFusion 55
//! scans it. Default writer properties, default SessionContext.
use std::sync::Arc;
use datafusion::arrow::array::{BinaryArray, RecordBatch};
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::common::ScalarValue;
use datafusion::datasource::file_format::parquet::ParquetFormat;
use datafusion::datasource::listing::{
ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl,
};
use datafusion::parquet::arrow::ArrowWriter;
use datafusion::prelude::*;
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
// One row group, one nullable Binary column, every value NULL —
// the footer statistics record null_count == num_rows, no min/max.
let schema = Arc::new(Schema::new(vec![Field::new(
"body",
DataType::Binary,
true,
)]));
let batch = RecordBatch::try_new(
schema.clone(),
vec![Arc::new(BinaryArray::from(vec![None::<&[u8]>; 100]))],
)
.unwrap();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("all_null.parquet");
let file = std::fs::File::create(&path).unwrap();
let mut w = ArrowWriter::try_new(file, schema, None).unwrap();
w.write(&batch).unwrap();
w.close().unwrap();
let ctx = SessionContext::new();
let url = ListingTableUrl::parse(format!("file://{}", path.display())).unwrap();
let options =
ListingOptions::new(Arc::new(ParquetFormat::default())).with_file_extension(".parquet");
let schema = options.infer_schema(&ctx.state(), &url).await?;
let table = ListingTable::try_new(
ListingTableConfig::new(url)
.with_listing_options(options)
.with_schema(schema),
)?;
// `body = X` can match nothing when every value is NULL, so the row
// group is prunable from statistics alone.
let df = ctx
.read_table(Arc::new(table))?
.filter(col("body").eq(lit(ScalarValue::Binary(Some(b"x".to_vec())))))?;
let plan = df.create_physical_plan().await?;
let batches = datafusion::physical_plan::collect(plan.clone(), ctx.task_ctx()).await?;
println!(
"rows = {} (correct on both versions)",
batches.iter().map(|b| b.num_rows()).sum::()
);
fn walk(p: &Arc) {
if let Some(m) = p.metrics() {
for metric in m.iter() {
if metric.value().name() == "row_groups_pruned_statistics" {
println!("{:?}", metric.value());
}
}
}
for c in p.children() {
walk(c);
}
}
walk(&plan);
Ok(())
}
```
## Expected behavior
The row group is pruned on 55 as it was on 54: the file's statistics prove `body = 'x'` cannot match (`null_count == row_count`), and the physical plan's `pruning_predicate` (identical on both versions in our larger application, including the `body_null_count != row_count` conjunct) already expresses that proof.
## Additional context
- Found upgrading a Parquet log store (ourios) from DF 54.0.0 → 55.0.0: three pruning-assertion tests went red with `pruned: 0` where 54 gave `pruned: 2`; written files byte-identical across the upgrade (parquet 58 vs 59 emit the same statistics for these columns), so this is read-path only.
- Triage hint: the regression reproduces through **`ListingTable` + the DataFrame API filter**. In our first attempt we could NOT reproduce via `register_parquet` + a SQL string (`WHERE body = X'6E6F7065'`) — that path does not prune on **either** 54 or 55 — so the SQL literal/rewrite path seems to sit on a different guarantee/pruning route and may mask the regression during triage.
- `EnabledStatistics::Page` vs `Chunk` and dictionary on/off for the column make no difference; defaults reproduce.
Contributor guide
Research direction
Start with the inline Cargo.toml and src/main.rs reproducer, running it against DataFusion 54 and 55 through ListingTable and the DataFrame filter. Trace the pruning_predicate and statistics row-group pruning path, including EnabledStatistics, and compare the reported row_groups_pruned_statistics metric. Done means the 55 path prunes the all-NULL row group as 54 does.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 63/100