apache / apache/iceberg

Spark: Implement SupportsReportOrdering in SparkBatchQueryScan

Open
#16,430 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

SparkBatchQueryScan stores sort_order_id per file in manifests but never implements SupportsReportOrdering, so BatchScanExec.outputOrdering always returns Nil

We can implement SupportsReportOrdering in SparkBatchQueryScan. Return the table's current SortOrder (converted via SortOrderToSpark) when all planned FileScanTasks share the same non-zero sort_order_id

This will benefit by Eliminating pre-sort in sort-merge joins, ordered aggregations, and MOR compaction reads when the table has a defined sort order and all files are sorted consistently.

```
CREATE TABLE db.events (user_id BIGINT, event_time TIMESTAMP)
USING iceberg WRITE ORDERED BY event_time;

INSERT INTO db.events SELECT * FROM source;

EXPLAIN SELECT * FROM db.events ORDER BY event_time;
-- Today: Sort[event_time] → BatchScanExec (outputOrdering=Nil)
-- After: BatchScanExec (outputOrdering=[event_time ASC]) — Sort eliminated
```

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by locating SparkBatchQueryScan and its use of FileScanTasks, then inspect SupportsReportOrdering, SortOrderToSpark, and BatchScanExec interactions described here. Done means outputOrdering reports the table SortOrder only when every planned task has the same non-zero sort_order_id, and the ordering is absent otherwise; verify the EXPLAIN example no longer adds Sort.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
57/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.