Missing `lower_bound`/`upper_bound` incorrectly treated as though all values for column in partition are NULL
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
Current, v0.4.0
### Describe the bug
The current implementation for partition filtering treats a missing `lower_bound`/`upper_bound` value as though all rows are null:
Current [manifest_evaluator.rs](https://github.com/apache/iceberg-rust/blob/50de31a5ef7518aeefad88ed9d815cd24ca962b8/crates/iceberg/src/expr/visitors/manifest_evaluator.rs#L156-L161):
```rs
fn less_than(
&mut self,
reference: &BoundReference,
datum: &Datum,
_predicate: &BoundPredicate,
) -> crate::Result {
let field = self.field_summary_for_reference(reference);
match &field.lower_bound {
Some(bound) if datum <= bound => ROWS_CANNOT_MATCH,
Some(_) => ROWS_MIGHT_MATCH,
None => ROWS_CANNOT_MATCH,
}
}
```
This means that if statistics were not computed on a given partition file, that file will be excluded no matter what.
For comparison, the [Java implementation](https://github.com/apache/iceberg/blob/a7f3dc79a2f42a4875ac35eec2137ecff15204fc/api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java#L210-L213) handles this correctly:
```java
T lower = lowerBound(term);
if (null == lower || NaNUtil.isNaN(lower)) {
// NaN indicates unreliable bounds. See the InclusiveMetricsEvaluator docs for more.
return ROWS_MIGHT_MATCH;
}
```
by treating a `null` lower bound as indicating that all rows might match.
### To Reproduce
_No response_
### Expected behavior
A partition file with a missing `lower_bound` column should _not be excluded_ (should be included) from scans that filter on that column with `<`/`<=`/`>`/`>=`.
### Willingness to contribute
~I can contribute a fix for this bug independently~
I no longer have bandwidth to contribute a fix
Contributor guide
Research direction
Start in crates/iceberg/src/expr/visitors/manifest_evaluator.rs around lines 156-161 and compare the Rust handling of missing bounds with the linked Java InclusiveMetricsEvaluator logic. Check the less-than and greater-than variants named by the issue, then verify that partitions with missing bounds are not excluded for <, <=, >, and >= filters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- Half a day
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100