apache / apache/datafusion

Find a way to communicate the ordering of a file back with the existing listing table implementation

Open
#13,891 5 comments 0 reactions 1 assignee Claimed by @zhuqi-lucas View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

This is the follow-up for:
https://github.com/apache/datafusion/pull/13874#pullrequestreview-2521214964

We add support (order by / sort) for DataFrameWriteOptions, but when a user try to query the table which the file already ordered, we can't get info from the table.

We need to find a way to communicate the ordering of a file back with the existing listing table implementation.

### Describe the solution you'd like

It is also conceivable that DataFusion itself could write custom metadata in paquet and other formats that support that custom metadata with the ordering, but that seems like we can use iceberg and other table formats.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.