apache / apache/iceberg-rust

feat(iceberg): thread per-file sort order into FileScanTask

Closed
#3,127 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

Part of https://github.com/apache/iceberg-rust/issues/3126

## Is your feature request related to a problem or challenge?

FileScanTask carries no sort-order information today. A file's sort_order_id is per-file (DataFile.sort_order_id), not table-wide: a table's default_sort_order() describes what new writes should conform to, not what any specific existing file is actually sorted by. Two files in the same manifest can even carry different sort_order_id values, if the table's sort order changed between when they were written.

Anything downstream that wants to know whether a set of files is safely treatable as sorted (the DataFusion side of #, or any other future consumer) needs that per-file signal resolved and available on FileScanTask. Today there's no way to get it without re-reading manifest data by hand.

##Describe the solution you'd like

Resolve each manifest entry's sort_order_id against the table's known sort orders (TableMetadata.sort_orders), and carry the result on a new FileScanTask.sort_order field. None means no sort order could be established, either because the file has no recorded sort_order_id, or because the id doesn't resolve against the table's known sort orders.

This threads through the existing PlanContext -> ManifestFileContext -> ManifestEntryContext pipeline in crates/iceberg/src/scan/, following the same pattern already used for unified_partition_type. No dependency on anything else, including the rest of #, so this is ready to implement now.

## Willingness to contribute:
I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/scan/ and trace the existing PlanContext -> ManifestFileContext -> ManifestEntryContext flow, using unified_partition_type as the reference pattern. Thread each manifest entry's resolved TableMetadata.sort_orders value into FileScanTask.sort_order, with None for missing or unresolved IDs, and verify the scan plan exposes the per-file result.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.