Find a way to communicate the ordering of a file back with the existing listing table implementation
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
This is the follow-up for:
https://github.com/apache/datafusion/pull/13874#pullrequestreview-2521214964
We add support (order by / sort) for DataFrameWriteOptions, but when a user try to query the table which the file already ordered, we can't get info from the table.
We need to find a way to communicate the ordering of a file back with the existing listing table implementation.
### Describe the solution you'd like
It is also conceivable that DataFusion itself could write custom metadata in paquet and other formats that support that custom metadata with the ordering, but that seems like we can use iceberg and other table formats.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.