apache / apache/parquet-java

Performance optimizations: Merged all LittleEndianDataInputStream functionality into ByteBufferInputStream

未关闭
#2,674 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Java Component: Parquet Priority: Major Type: enhancement
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

This PR is all performance optimization. In benchmarking with Trino, we find query performance to improve from 5% to 15%, depending on the query, and that includes all the I/O time from S3.

The main modification is to merge all of LittleEndianDataInputStream functionality into ByteBufferInputStream, which yields the following benefits:
- Elimination of extra layers of abstraction and method call overhead
- Enable the use of intrinsics for readInt, readLong, etc.
- Availability of faster access methods like readFully and skipFully, without the need for helper functions
- Reduces some object creation in the performance critical path

This also includes and enables performance optimizations to:
- ByteBitPackingValuesReader
- PlainValuesReader
- RunLengthBitPackingHybridDecoder

Context:
I've been working on improving Parquet reading performance in Trino, mostly by profiling while running performance benchmarks and TPCDS queries. This PR is a subset of the changes I made that have more than doubled the performance of a lot of TPCDS queries (wall clock time, including the S3 access time). If you are kind enough to accept these changes, I have more I would like to contribute.

**Reporter**: [Timothy Miller](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=theosib-amazon) / @theosib-amazon
#### Externally tracked issue: [https://github.com/apache/parquet-mr/pull/953](https://github.com/apache/parquet-mr/pull/953)

**Note**: *This issue was originally created as [PARQUET-2135](https://issues.apache.org/jira/browse/PARQUET-2135). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先检查外部跟踪的 pull request 953 以及指定的组件:ByteBufferInputStream、LittleEndianDataInputStream、ByteBitPackingValuesReader、PlainValuesReader 和 RunLengthBitPackingHybridDecoder。比较相关的 Trino 和 TPCDS 基准测试结果;合并的功能和优化经过验证且没有回归,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering, performance
Issue 类型
重构
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。