Prune full-data-file Iceberg changelog delete tasks by metadata predicates
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 22
Description
**Is your feature request related to a problem? Please describe.**
Auron already supports native execution of `DeletedDataFileScanTask` when `existingDeletes()` is empty. However, `filterChangelogTasks` evaluates metadata predicates only for `AddedRowsScanTask` and retains all other tasks.
As a result, a supported full-data-file DELETE task remains in the native scan plan even when its metadata proves that it cannot match the query. See the [current task filtering implementation](https://github.com/apache/auron/blob/04eeb90d2911465eb350253eacc281a16bb5f37f/thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala#L614-L634).
For example, given a changelog range containing only full-data-file DELETE tasks:
```sql
SELECT id
FROM changes
WHERE _change_type = 'INSERT';
```
The result is empty, but the DELETE tasks are retained. The existing integration test explicitly checks this behavior and asserts `numFiles == 1` after executing the query. See the [existing regression test](https://github.com/apache/auron/blob/04eeb90d2911465eb350253eacc281a16bb5f37f/thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala#L810-L821).
The same limitation applies when `_commit_snapshot_id` or `_change_ordinal` excludes a DELETE task. This leaves unnecessary scan work in metadata-filtered changelog queries.
**Describe the solution you'd like**
Extend the existing changelog metadata pruning to supported `DeletedDataFileScanTask` instances with empty `existingDeletes()`.
Reuse the current per-task metadata values and predicate evaluation for:
- `_change_type`
- `_commit_snapshot_id`
- `_change_ordinal`
Support the existing `=` and `IN` predicates combined with `AND`, and retain the complete post-scan filter.
Apply pruning after the existing native task eligibility checks. Preserve the expression-ID checks, operator boundaries, runtime-filter tag propagation, and fallback behavior for unsupported tasks.
This change should not introduce position/equality delete handling or extend predicate extraction to `OR`, `NOT`, or mixed metadata/data-column conditions.
Validation should cover:
- A DELETE-only range filtered by `_change_type = 'INSERT'`: empty results and zero retained native scan tasks.
- A matching DELETE predicate: matching tasks and rows remain.
- DELETE tasks from different snapshots: equality, `IN`, and `AND` predicates on snapshot IDs and change ordinals retain only matching tasks.
- A range containing both INSERT and full-data-file DELETE tasks: either task type can be excluded independently.
- A predicate that excludes every task: execution succeeds with empty results.
- Unsupported predicates and tasks: existing behavior is preserved.
Check explicit expected rows and compare Auron-enabled execution with Spark. For eligible cases, assert native scan conversion and inspect `scanTasks` to verify pruning; result equality alone does not demonstrate that scan work was reduced.
**Describe alternatives you've considered**
Continue relying on the post-scan filter. This preserves query results, but retains DELETE tasks that can already be excluded using task-level metadata.
**Additional context**
This is a follow-up to:
- [#2435](https://github.com/apache/auron/pull/2435), which added native full-data-file DELETE changelog scans.
- [#2456](https://github.com/apache/auron/pull/2456), which added metadata pruning for INSERT tasks and explicitly retained non-`AddedRowsScanTask` tasks.
The expected implementation scope is `IcebergScanSupport.scala` and focused regression coverage in `AuronIcebergIntegrationSuite.scala`. No new configuration or public API is needed.
This proposal is based on source inspection and existing test assertions; no new reproduction or performance benchmark has been run.
Contributor guide
Research direction
Start in thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala, reading filterChangelogTasks and its existing native task eligibility checks. Run the focused cases in thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala around the existing regression test. Done means supported full-data-file DELETE tasks are pruned by the listed metadata predicates, while results, fallback behavior, and unsupported cases remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100