apache / apache/arrow

[C++][Parquet] Allow Truncate min-max Statistics

Open
#36,139 3 comments 1 reaction 0 assignees View on GitHub
Component: C++ Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Currently, in parquet c++, if the min-max is string / binary:
1. During writing values, min-max will be collect
2. When building statistics, there is apply truncate, which will discard min-max if they're longer than expected

Can we use a "truncate" for this:
1. For BINARY
1. If it's minimum, just truncate to length is ok
2. if it's maximum:
1. if the truncated binary would be 0xFF 0xFF ... 0xFF, we cannot truncate it
2. Otherwise, get the "next" valid truncated binary
2. For String
1. If it's minimum, just truncate and to **a valid utf8** is ok
2. If it's maximum, first truncate to a valid utf8, then try to advance it.

References:
1. https://github.com/apache/parquet-mr/pull/481
2. https://github.com/apache/arrow-rs/pull/4389

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.