apache / apache/datafusion-comet
Iceberg scan falls back to Spark on IS NULL/IS NOT NULL over list/map columns (stale complex-type check)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
## Describe the bug
`CometScanRule` declines every Iceberg scan whose pushed filters contain an `IS NULL`/`IS NOT NULL` predicate on a complex-typed column, with the reason:
```
IS NULL / IS NOT NULL predicates on complex type columns (struct/array/map) are not yet supported by iceberg-rust
```
The check (added in #2528) predates the current iceberg-rust pin (bumped in #5651). iceberg-rust's Arrow predicate visitor now evaluates `is_null`/`is_not_null` on **list** and **map** columns through arrow's native `is_null`/`is_not_null` (`crates/iceberg/src/arrow/reader/predicate_visitor.rs`); only **struct** columns are rejected (`project_column` errors on `DataType::Struct`).
## Impact
The check is a false positive for list columns, which is a common real pattern: Spark's optimizer pushes `isnotnull(arr)` below `Generate` for `explode(arr)`:
```
BatchScan local_tpch.orders_nested [COMET: IS NULL / IS NOT NULL predicates on complex type columns ...]
```
Each declined scan forces the whole plan back to Spark and cascades into further fallbacks (`Comet aggregate that merges intermediate buffers requires a Comet child aggregate ...`). On a 24-query derived TPC-H benchmark (SF1, 3 runs, cold caches, Spark 4.1.3, Iceberg hadoop catalog, table with `array` line-item column):
| metric | check active | list false-positive removed |
|---|---:|---:|
| Comet/Iceberg total (24q) | 20.63 s | 16.93 s (−17.9%) |
| `q1_nested_explode` | 2.20 s | 0.77 s (−65%) |
| `e14_nested` | 2.03 s | 0.86 s (−58%) |
| Comet/Iceberg vs Comet/Parquet total | +26% | +3.7% |
Parity vs vanilla Spark verified on all 24 queries in both configurations.
A fix that narrows the check to struct-only columns will be submitted as a PR.
Contributor guide
Research direction
Start in CometScanRule, then read crates/iceberg/src/arrow/reader/predicate_visitor.rs to compare the supported null predicates for list, map, and struct columns. Run the affected Iceberg scan coverage or reproduction described in the issue; done means list/map scans no longer fall back to Spark while struct-column predicates remain rejected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100