apache / apache/parquet-format

Add SortingColumn on a file level

Open
#501 3 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

### Describe the enhancement requested

Currently, the `sorting_columns` is only defined on the `RowGroupMetadata`. This makes it hard to get the sort status of the entire file when it has more than one row group.

This makes it generally very hard to utilize during query optimization.

Is there any interest in adding `sorting_columns` to the `FileMetadata` optionally? I feel like this is more generally useful than having it on the `row group` metadata. I would be happy to draft up a proposal.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the existing sorting_columns definition on RowGroupMetadata with the proposed optional field on FileMetadata. Review how file and row-group metadata are represented in the Parquet format, then determine the proposal and compatibility details needed for a file-level sort status; done means the project has an agreed format change.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.