Parquet: notNaN filter crashes on columns absent from older files
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Other
### Please describe the bug 🐞
## Summary
A Parquet read using a `notNaN` filter crashes when the referenced float or double column exists in the projected Iceberg schema but is absent from an older Parquet file.
I reproduced this through Iceberg's generic Java Parquet reader rather than a specific query engine. Spark and Flink can also produce this predicate when translating comparisons against NaN.
This is a normal schema-evolution scenario:
1. Write a Parquet file with schema `{ required long id }`.
2. Add an optional float column such as `added_float`.
3. Read the old file using the evolved schema and filter with `notNaN("added_float")`.
The added column should read as null for old rows. Iceberg defines null as not NaN, so the row group must remain readable. Instead, reader initialization throws a `NullPointerException`.
I reproduced this on `main` at commit:
```text
1ec15051dc09ef6a0d61fa332eb8f2503f3170f8
```
The same faulty code is present in the `apache-iceberg-1.11.0` tag.
## Minimal reproduction
In `TestDictionaryRowGroupFilter#testColumnNotInFile`, add `notNaN("not_in_file")` to the existing `exprs` array:
```java
Expression[] exprs =
new Expression[] {
lessThan("not_in_file", 1.0f),
lessThanOrEqual("not_in_file", 1.0f),
equal("not_in_file", 1.0f),
greaterThan("not_in_file", 1.0f),
greaterThanOrEqual("not_in_file", 1.0f),
notNull("not_in_file"),
isNull("not_in_file"),
notEqual("not_in_file", 1.0f),
notNaN("not_in_file")
};
```
Then run:
```shell
./gradlew --no-build-cache \
:iceberg-parquet:test \
--tests org.apache.iceberg.parquet.TestDictionaryRowGroupFilter \
--no-daemon
```
I also reproduced the failure twice through the production `Parquet.read(...)` path by writing an old-schema file and reading it with the evolved schema.
## Expected behavior
The reader should retain the row group. Because the column is absent from the file, its values are null, and null values satisfy Iceberg's `notNaN` expression semantics.
## Actual behavior
The read fails with:
```text
Cannot invoke "java.lang.Boolean.booleanValue()" because
the return value of "java.util.Map.get(Object)" is null
```
The exception originates from `ParquetDictionaryRowGroupFilter.java:178` and is reached through `ReadConf.java:109`.
## Root cause
`ParquetDictionaryRowGroupFilter.EvalVisitor#notNaN` evaluates:
```java
if (mayContainNulls.get(id)) {
return ROWS_MIGHT_MATCH;
}
```
before checking whether the field ID exists in the file metadata:
```java
Boolean hasNonDictPage = isFallback.get(id);
if (hasNonDictPage == null || hasNonDictPage) {
return ROWS_MIGHT_MATCH;
}
```
For a column absent from the file, neither map has an entry for the field ID. The first lookup returns null and is unboxed as a boolean, causing the exception.
Moving the existing absent/fallback-column guard before the nullability lookup should preserve behavior for present columns while conservatively retaining row groups for absent columns.
## Related history
- #6431 added the nullability check for physically present columns containing nulls, but did not cover columns absent from the file.
- #16692 addresses an initial-default problem in `ParquetMetricsRowGroupFilter` and does not change this dictionary-filter path.
I searched existing open and closed issues and pull requests using combinations of `notNaN`, `ParquetDictionaryRowGroupFilter`, `missing column`, `column not in file`, and `schema evolution`, and did not find an active fix for this crash.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with ParquetDictionaryRowGroupFilter.java, especially EvalVisitor#notNaN and the logic around line 178, then review TestDictionaryRowGroupFilter#testColumnNotInFile. Run the targeted Gradle test command from the issue and confirm that notNaN on an absent column retains the row group without an exception, while existing-column behavior remains covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 85/100