apache / apache/iceberg-rust

Missing `lower_bound`/`upper_bound` incorrectly treated as though all values for column in partition are NULL

Open Beginner friendly
#1,354 6 comments 0 reactions 0 assignees View on GitHub
bug stale
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

Current, v0.4.0

### Describe the bug

The current implementation for partition filtering treats a missing `lower_bound`/`upper_bound` value as though all rows are null:

Current [manifest_evaluator.rs](https://github.com/apache/iceberg-rust/blob/50de31a5ef7518aeefad88ed9d815cd24ca962b8/crates/iceberg/src/expr/visitors/manifest_evaluator.rs#L156-L161):
```rs
fn less_than(
&mut self,
reference: &BoundReference,
datum: &Datum,
_predicate: &BoundPredicate,
) -> crate::Result {
let field = self.field_summary_for_reference(reference);
match &field.lower_bound {
Some(bound) if datum <= bound => ROWS_CANNOT_MATCH,
Some(_) => ROWS_MIGHT_MATCH,
None => ROWS_CANNOT_MATCH,
}
}
```

This means that if statistics were not computed on a given partition file, that file will be excluded no matter what.

For comparison, the [Java implementation](https://github.com/apache/iceberg/blob/a7f3dc79a2f42a4875ac35eec2137ecff15204fc/api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java#L210-L213) handles this correctly:

```java
T lower = lowerBound(term);
if (null == lower || NaNUtil.isNaN(lower)) {
// NaN indicates unreliable bounds. See the InclusiveMetricsEvaluator docs for more.
return ROWS_MIGHT_MATCH;
}
```

by treating a `null` lower bound as indicating that all rows might match.

### To Reproduce

_No response_

### Expected behavior

A partition file with a missing `lower_bound` column should _not be excluded_ (should be included) from scans that filter on that column with `<`/`<=`/`>`/`>=`.

### Willingness to contribute

~I can contribute a fix for this bug independently~

I no longer have bandwidth to contribute a fix

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/expr/visitors/manifest_evaluator.rs around lines 156-161 and compare the Rust handling of missing bounds with the linked Java InclusiveMetricsEvaluator logic. Check the less-than and greater-than variants named by the issue, then verify that partitions with missing bounds are not excluded for <, <=, >, and >= filters.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
Half a day
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.