apache / apache/datafusion-comet

Sort-aware Iceberg reads: per-partition streaming merge + report ordering and grouping (v1)

Open
#5,337 0 comments 0 reactions 1 assignee Claimed by @parthchandra View on GitHub
area:scan enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

The native Iceberg scan now preserves the sort order instead of throwing it away. Within each Spark partition it reads each already-sorted file as its own stream and k-way-merges them into one sorted stream, then tells Spark the output is sorted so redundant sorts can be dropped. It also reports Iceberg's key-grouped partitioning so
storage-partitioned joins can skip the shuffle.

Scope of this first version:
- Ordering reporting is limited to plain-column (identity) sort keys that are in the projection. Anything else falls back to an unordered read, so it's always correct.
- Two flags: spark.comet.scan.icebergNative.sortMerge.enabled (on by default) and spark.comet.scan.icebergNative.reportPartitioning.enabled (off by default).
- No changes to iceberg-rust.

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.