Core, Parquet: Incorrect null counting when null_count is missing from Parquet stats
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
None
### Please describe the bug 🐞
Parquet's Statistics#getNumNulls returns -1 when null_count is missing from the stats. But our current metrics logic doesn't handle this -1 value specifically, instead it just adds to the existing count.
For a single row group the -1 total was dropped because Metrics only keeps non-negative counts. But say we have row group 0 that has one null value and row group 1 doesn't have null_count stats and return -1, we end up counting total null count as 0, which is wrong. A query engine that relays on null count can skip the file completely when evaluating predicate like WHERE c IS NULL causing wrong results.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
No file or test is named. Start by tracing the core metrics aggregation from Parquet Statistics#getNumNulls and inspect how Metrics combines row-group values; reproduce the two-row-group case, then add regression coverage showing that missing null_count cannot reduce a known null total and verify the IS NULL scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100