apache / apache/parquet-format
Add SortingColumn on a file level
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
### Describe the enhancement requested
Currently, the `sorting_columns` is only defined on the `RowGroupMetadata`. This makes it hard to get the sort status of the entire file when it has more than one row group.
This makes it generally very hard to utilize during query optimization.
Is there any interest in adding `sorting_columns` to the `FileMetadata` optionally? I feel like this is more generally useful than having it on the `row group` metadata. I would be happy to draft up a proposal.
Contributor guide
Research direction
Start by comparing the existing sorting_columns definition on RowGroupMetadata with the proposed optional field on FileMetadata. Review how file and row-group metadata are represented in the Parquet format, then determine the proposal and compatibility details needed for a file-level sort status; done means the project has an agreed format change.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100