Parquet's statistics' min/max value may be encoded in big endian
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Describe the bug**
As per [Parquet's spec](https://parquet.apache.org/docs/file-format/data-pages/encodings/) and [Java implementation](https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/statistics/IntStatistics.java#L82), Statistics use plain encoding which encodes INT64 to bytes in litten endian.
In arrow's official Parquet implementation, when decoding column statistics, it decodes data in little endian:
https://github.com/apache/arrow-rs/blob/3788fd20f053ee58f08b4d09cd4dac5bb9b96c06/parquet/src/file/statistics.rs#L171-L177
But when writing min/max value of statistics, it simply convert the memory representation of i64 values into byte slice, which is platform dependent.
https://github.com/apache/arrow-rs/blob/3788fd20f053ee58f08b4d09cd4dac5bb9b96c06/parquet/src/data_type.rs#L451-L463
**To Reproduce**
It would be rather easy to reproduce this problem, but I don't have any big endian device like MIPS server by my side.
**Expected behavior**
Encode min/max value of statistics into little endian bytes.
**Additional context**
When encoding stats, Parquet uses `AsBytes` trait to convert i64 into byte slice,
https://github.com/apache/arrow-rs/blob/3788fd20f053ee58f08b4d09cd4dac5bb9b96c06/parquet/src/data_type.rs#L428-L431
Thus the lifetime of slice returned is bound with the value itself. If we want to convert a i64 into little endian byte slice in a big endian platform, we must create a temporarily array to store the converted little endian bytes of the value instead of just reinterpret the address of value into a byte slice. When `as_bytes` method returns, the temp array will be dropped which violates the trait's lifetime constraint. We may need to change `AsBytes` into sth like:
```rs
pub trait AsBytes {
fn encode(&self, buf: &mut Vec) -> usize;
}
```
Contributor guide
Research direction
Start by reading parquet/src/file/statistics.rs for statistics decoding and parquet/src/data_type.rs for AsBytes and min/max encoding. Trace how integer statistics are serialized and determine how the implementation can preserve the required byte order and lifetimes. Done means encoded INT64 statistics are little endian on all platforms, including big endian systems.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100