apache / apache/datafusion-comet
Sort-aware Iceberg reads: per-partition streaming merge + report ordering and grouping (v1)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
The native Iceberg scan now preserves the sort order instead of throwing it away. Within each Spark partition it reads each already-sorted file as its own stream and k-way-merges them into one sorted stream, then tells Spark the output is sorted so redundant sorts can be dropped. It also reports Iceberg's key-grouped partitioning so
storage-partitioned joins can skip the shuffle.
Scope of this first version:
- Ordering reporting is limited to plain-column (identity) sort keys that are in the projection. Anything else falls back to an unordered read, so it's always correct.
- Two flags: spark.comet.scan.icebergNative.sortMerge.enabled (on by default) and spark.comet.scan.icebergNative.reportPartitioning.enabled (off by default).
- No changes to iceberg-rust.
### Describe the potential solution
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.