apache / apache/iceberg-python

Visitor and evaluator edge cases can over-prune files or mishandle nulls

Open
#3,498 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

Several visitor/evaluator edge cases appear unsafe or inconsistent:

1. `_StrictMetricsEvaluator.visit_not_equal` / `visit_not_in` return `ROWS_MUST_MATCH` when a file can contain nulls or NaNs. Example stats with `[null, 5]` or `[NaN, 5.0]` and lower/upper bounds both `5` return true for `NotEqualTo("x", 5)` / `NotIn("x", {5})`, even though one row does not match. This can incorrectly mark whole files deleted.

2. `_StrictMetricsEvaluator.eval` returns `ROWS_MUST_MATCH` for `record_count <= 0`. `record_count=0` is vacuously true, but `record_count=-1` is unknown per the local comment; even `AlwaysFalse()` returns true.

3. `ResidualVisitor` comparison methods directly compare partition values to literals. A nullable identity partition value of `None` with `LessThan("x", 1)` raises `TypeError`, while row evaluation returns false.

4. `ResidualVisitor.visit_not_nan(None)` returns `AlwaysFalse`, while expression evaluation treats `NotNaN(None)` as true. Existing tests encode both behaviors, so the semantics are inconsistent.

Validated against the current tree; examples use stats/partition shapes already supported by the repo tests.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.