apache / apache/iceberg

Core, Parquet: Incorrect null counting when null_count is missing from Parquet stats

Open
#17,558 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

None

### Please describe the bug 🐞

Parquet's Statistics#getNumNulls returns -1 when null_count is missing from the stats. But our current metrics logic doesn't handle this -1 value specifically, instead it just adds to the existing count.

For a single row group the -1 total was dropped because Metrics only keeps non-negative counts. But say we have row group 0 that has one null value and row group 1 doesn't have null_count stats and return -1, we end up counting total null count as 0, which is wrong. A query engine that relays on null count can skip the file completely when evaluating predicate like WHERE c IS NULL causing wrong results.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by tracing the core metrics aggregation from Parquet Statistics#getNumNulls and inspect how Metrics combines row-group values; reproduce the two-row-group case, then add regression coverage showing that missing null_count cannot reduce a known null total and verify the IS NULL scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.