apache / apache/parquet-format

Discussion around storing SUM as a optional metadata field.

Open
#596 1 comment 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

### Describe the enhancement requested

Currently parquet only stores fields for predicate pushdown for scan pruning, but we can also keep `sum` in metadata field, while this won't help in predicate pushdown, but it can act as a pre-aggregation for sum and avg queries.

While this may be challenging and bit complex because we would need to handle sum/overflow etc. But this can be optional just like bloom filters and would be helpful for many usecases. And shouldn't have much of a overhead due to it being a stateless operation.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing Parquet metadata fields and optional bloom-filter handling described in the issue. Define how optional SUM metadata would be represented and how overflow and SUM/AVG pre-aggregation should work. Done means the format design and its trade-offs are agreed.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.