apache / apache/iceberg

Parquet: notNaN filter crashes on columns absent from older files

Open Beginner friendly
#17,301 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Other

### Please describe the bug 🐞

## Summary

A Parquet read using a `notNaN` filter crashes when the referenced float or double column exists in the projected Iceberg schema but is absent from an older Parquet file.

I reproduced this through Iceberg's generic Java Parquet reader rather than a specific query engine. Spark and Flink can also produce this predicate when translating comparisons against NaN.

This is a normal schema-evolution scenario:

1. Write a Parquet file with schema `{ required long id }`.
2. Add an optional float column such as `added_float`.
3. Read the old file using the evolved schema and filter with `notNaN("added_float")`.

The added column should read as null for old rows. Iceberg defines null as not NaN, so the row group must remain readable. Instead, reader initialization throws a `NullPointerException`.

I reproduced this on `main` at commit:

```text
1ec15051dc09ef6a0d61fa332eb8f2503f3170f8
```

The same faulty code is present in the `apache-iceberg-1.11.0` tag.

## Minimal reproduction

In `TestDictionaryRowGroupFilter#testColumnNotInFile`, add `notNaN("not_in_file")` to the existing `exprs` array:

```java
Expression[] exprs =
new Expression[] {
lessThan("not_in_file", 1.0f),
lessThanOrEqual("not_in_file", 1.0f),
equal("not_in_file", 1.0f),
greaterThan("not_in_file", 1.0f),
greaterThanOrEqual("not_in_file", 1.0f),
notNull("not_in_file"),
isNull("not_in_file"),
notEqual("not_in_file", 1.0f),
notNaN("not_in_file")
};
```

Then run:

```shell
./gradlew --no-build-cache \
:iceberg-parquet:test \
--tests org.apache.iceberg.parquet.TestDictionaryRowGroupFilter \
--no-daemon
```

I also reproduced the failure twice through the production `Parquet.read(...)` path by writing an old-schema file and reading it with the evolved schema.

## Expected behavior

The reader should retain the row group. Because the column is absent from the file, its values are null, and null values satisfy Iceberg's `notNaN` expression semantics.

## Actual behavior

The read fails with:

```text
Cannot invoke "java.lang.Boolean.booleanValue()" because
the return value of "java.util.Map.get(Object)" is null
```

The exception originates from `ParquetDictionaryRowGroupFilter.java:178` and is reached through `ReadConf.java:109`.

## Root cause

`ParquetDictionaryRowGroupFilter.EvalVisitor#notNaN` evaluates:

```java
if (mayContainNulls.get(id)) {
return ROWS_MIGHT_MATCH;
}
```

before checking whether the field ID exists in the file metadata:

```java
Boolean hasNonDictPage = isFallback.get(id);
if (hasNonDictPage == null || hasNonDictPage) {
return ROWS_MIGHT_MATCH;
}
```

For a column absent from the file, neither map has an entry for the field ID. The first lookup returns null and is unboxed as a boolean, causing the exception.

Moving the existing absent/fallback-column guard before the nullability lookup should preserve behavior for present columns while conservatively retaining row groups for absent columns.

## Related history

- #6431 added the nullability check for physically present columns containing nulls, but did not cover columns absent from the file.
- #16692 addresses an initial-default problem in `ParquetMetricsRowGroupFilter` and does not change this dictionary-filter path.

I searched existing open and closed issues and pull requests using combinations of `notNaN`, `ParquetDictionaryRowGroupFilter`, `missing column`, `column not in file`, and `schema evolution`, and did not find an active fix for this crash.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with ParquetDictionaryRowGroupFilter.java, especially EvalVisitor#notNaN and the logic around line 178, then review TestDictionaryRowGroupFilter#testColumnNotInFile. Run the targeted Gradle test command from the issue and confirm that notNaN on an absent column retains the row group without an exception, while existing-column behavior remains covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.