apache / apache/iceberg-rust

[Epic] Sort-order reporting for iceberg-datafusion scans

Open
#3,126 2 comments 0 reactions 0 assignees View on GitHub
epic
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### What's the feature are you trying to implement?

Today, `IcebergTableScan::compute_properties` always builds `EquivalenceProperties` with no ordering, so DataFusion has no way to know a scan's output is already sorted. DataFusion's optimizer decides whether to insert a sort by checking what each node's `PlanProperties::output_ordering` already claims. A sort-merge join, an ORDER BY, or a MERGE all require their input in a specific order; if the node feeding them already claims that order, the physical optimizer can skip the SortExec (or SortPreservingMergeExec) it would otherwise insert above the scan.

This epic tracks the iceberg-rust side, split into two parts that land independently:

### Part 1: Per-file sort order on FileScanTask

FileScanTask carries no sort-order information today. A file's sort_order_id is per-file (`DataFile.sort_order_id`), not table-wide — a table's `default_sort_order()` describes what new writes should conform to, not what any specific existing file actually is sorted by. Resolve each manifest entry's sort_order_id against the table's known sort orders and carry the result on FileScanTask, threaded through PlanContext -> ManifestFileContext -> ManifestEntryContext

### Part 2: DataFusion integration

Depends on apache/iceberg-rust#2671 (eager scan planning) merging first.

`PlanProperties::output_ordering` is one value for the whole IcebergTableScan node, not one per partition, so an ordering can only be claimed when every file across every one of #2671's task groups agrees on the same sort order (via Part 1's per-file data). When they do, the claim must be backed by a real per-partition merge in execute() (e.g. DataFusion's `StreamingMergeBuilder`), since two files agreeing on `sort_order_id` doesn't mean their key ranges don't overlap. When even one file disagrees or has no resolvable sort order, fall back to today's behavior exactly.

Related
* apache/iceberg#16430 — Java SupportsReportOrdering
* apache/datafusion-comet#4367 — Comet outputOrdering forwarding
* apache/iceberg-rust#2671 — eager scan planning (Part 2's dependency)

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start at IcebergTableScan::compute_properties and trace FileScanTask through PlanContext, ManifestFileContext, and ManifestEntryContext. Review the eager scan planning dependency in apache/iceberg-rust#2671 before working on the two independently landing parts. Done means per-file sort orders are carried and DataFusion reports ordering only when all tasks agree and execute() performs the required merge, preserving today’s fallback otherwise.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.