[C++][Parquet] Allow Truncate min-max Statistics
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Currently, in parquet c++, if the min-max is string / binary:
1. During writing values, min-max will be collect
2. When building statistics, there is apply truncate, which will discard min-max if they're longer than expected
Can we use a "truncate" for this:
1. For BINARY
1. If it's minimum, just truncate to length is ok
2. if it's maximum:
1. if the truncated binary would be 0xFF 0xFF ... 0xFF, we cannot truncate it
2. Otherwise, get the "next" valid truncated binary
2. For String
1. If it's minimum, just truncate and to **a valid utf8** is ok
2. If it's maximum, first truncate to a valid utf8, then try to advance it.
References:
1. https://github.com/apache/parquet-mr/pull/481
2. https://github.com/apache/arrow-rs/pull/4389
### Component(s)
C++, Parquet
Contributor guide
Assessment
This issue has not been assessed yet.