apache / apache/iceberg

Core, Parquet: Null counts are incorrect for child fields under a null struct

Open
#17,561 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

None

### Query engine

None

### Please describe the bug 🐞

When an optional struct is null, OptionWriter writes a null directly to every leaf column it contains, so the writers for those columns never see the value and cannot count it. OptionWriter dropped its own null count in that case, with a comment saying nested null stats were not used. They are used: the counts it returns become DataFile.nullValueCounts.

Float and double are the only types whose writers report metrics, and ParquetMetrics prefers writer metrics over footer statistics, so the correct footer count was never used. A float or double under a nullable struct was reported as having 0 nulls even when the struct was null for some rows.

The incorrect counting of nulls could affect query engines that relay on this stats for optimization. For example, they could simply skip the file with null_count == 0 for predicate WHERE c.f_id IS NULL and produces wrong result.

Fix by PR: https://github.com/apache/iceberg/pull/17560

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with OptionWriter, ParquetMetrics, and DataFile.nullValueCounts to trace how null counts are produced for float and double fields under a null struct. Compare the behavior with linked pull request #17560 and verify that affected Parquet metrics report the correct null counts without relying on footer statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.