apache / apache/auron

Prune full-data-file Iceberg changelog delete tasks by metadata predicates

Closed
#2,511 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 14h
Merged PRs (30d)
22

Description

**Is your feature request related to a problem? Please describe.**

Auron already supports native execution of `DeletedDataFileScanTask` when `existingDeletes()` is empty. However, `filterChangelogTasks` evaluates metadata predicates only for `AddedRowsScanTask` and retains all other tasks.

As a result, a supported full-data-file DELETE task remains in the native scan plan even when its metadata proves that it cannot match the query. See the [current task filtering implementation](https://github.com/apache/auron/blob/04eeb90d2911465eb350253eacc281a16bb5f37f/thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala#L614-L634).

For example, given a changelog range containing only full-data-file DELETE tasks:

```sql
SELECT id
FROM changes
WHERE _change_type = 'INSERT';
```

The result is empty, but the DELETE tasks are retained. The existing integration test explicitly checks this behavior and asserts `numFiles == 1` after executing the query. See the [existing regression test](https://github.com/apache/auron/blob/04eeb90d2911465eb350253eacc281a16bb5f37f/thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala#L810-L821).

The same limitation applies when `_commit_snapshot_id` or `_change_ordinal` excludes a DELETE task. This leaves unnecessary scan work in metadata-filtered changelog queries.

**Describe the solution you'd like**

Extend the existing changelog metadata pruning to supported `DeletedDataFileScanTask` instances with empty `existingDeletes()`.

Reuse the current per-task metadata values and predicate evaluation for:

- `_change_type`
- `_commit_snapshot_id`
- `_change_ordinal`

Support the existing `=` and `IN` predicates combined with `AND`, and retain the complete post-scan filter.

Apply pruning after the existing native task eligibility checks. Preserve the expression-ID checks, operator boundaries, runtime-filter tag propagation, and fallback behavior for unsupported tasks.

This change should not introduce position/equality delete handling or extend predicate extraction to `OR`, `NOT`, or mixed metadata/data-column conditions.

Validation should cover:

- A DELETE-only range filtered by `_change_type = 'INSERT'`: empty results and zero retained native scan tasks.
- A matching DELETE predicate: matching tasks and rows remain.
- DELETE tasks from different snapshots: equality, `IN`, and `AND` predicates on snapshot IDs and change ordinals retain only matching tasks.
- A range containing both INSERT and full-data-file DELETE tasks: either task type can be excluded independently.
- A predicate that excludes every task: execution succeeds with empty results.
- Unsupported predicates and tasks: existing behavior is preserved.

Check explicit expected rows and compare Auron-enabled execution with Spark. For eligible cases, assert native scan conversion and inspect `scanTasks` to verify pruning; result equality alone does not demonstrate that scan work was reduced.

**Describe alternatives you've considered**

Continue relying on the post-scan filter. This preserves query results, but retains DELETE tasks that can already be excluded using task-level metadata.

**Additional context**

This is a follow-up to:

- [#2435](https://github.com/apache/auron/pull/2435), which added native full-data-file DELETE changelog scans.
- [#2456](https://github.com/apache/auron/pull/2456), which added metadata pruning for INSERT tasks and explicitly retained non-`AddedRowsScanTask` tasks.

The expected implementation scope is `IcebergScanSupport.scala` and focused regression coverage in `AuronIcebergIntegrationSuite.scala`. No new configuration or public API is needed.

This proposal is based on source inspection and existing test assertions; no new reproduction or performance benchmark has been run.

Contributor guide

Open the contributing guide

Research direction

Start in thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala, reading filterChangelogTasks and its existing native task eligibility checks. Run the focused cases in thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala around the existing regression test. Done means supported full-data-file DELETE tasks are pruned by the listed metadata predicates, while results, fallback behavior, and unsupported cases remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.