feat(iceberg): thread per-file sort order into FileScanTask
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
Part of https://github.com/apache/iceberg-rust/issues/3126
## Is your feature request related to a problem or challenge?
FileScanTask carries no sort-order information today. A file's sort_order_id is per-file (DataFile.sort_order_id), not table-wide: a table's default_sort_order() describes what new writes should conform to, not what any specific existing file is actually sorted by. Two files in the same manifest can even carry different sort_order_id values, if the table's sort order changed between when they were written.
Anything downstream that wants to know whether a set of files is safely treatable as sorted (the DataFusion side of #, or any other future consumer) needs that per-file signal resolved and available on FileScanTask. Today there's no way to get it without re-reading manifest data by hand.
##Describe the solution you'd like
Resolve each manifest entry's sort_order_id against the table's known sort orders (TableMetadata.sort_orders), and carry the result on a new FileScanTask.sort_order field. None means no sort order could be established, either because the file has no recorded sort_order_id, or because the id doesn't resolve against the table's known sort orders.
This threads through the existing PlanContext -> ManifestFileContext -> ManifestEntryContext pipeline in crates/iceberg/src/scan/, following the same pattern already used for unified_partition_type. No dependency on anything else, including the rest of #, so this is ready to implement now.
## Willingness to contribute:
I can contribute to this feature independently
Contributor guide
Research direction
Start in crates/iceberg/src/scan/ and trace the existing PlanContext -> ManifestFileContext -> ManifestEntryContext flow, using unified_partition_type as the reference pattern. Thread each manifest entry's resolved TableMetadata.sort_orders value into FileScanTask.sort_order, with None for missing or unresolved IDs, and verify the scan plan exposes the per-file result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100