Performance optimizations: Merged all LittleEndianDataInputStream functionality into ByteBufferInputStream
- 主要言語
- Java
- スター
- 3.1k
- フォーク
- 1.6k
- 平均マージ
- 3日 12時間
- マージ済み PR(30日)
- 33
説明
This PR is all performance optimization. In benchmarking with Trino, we find query performance to improve from 5% to 15%, depending on the query, and that includes all the I/O time from S3.
The main modification is to merge all of LittleEndianDataInputStream functionality into ByteBufferInputStream, which yields the following benefits:
- Elimination of extra layers of abstraction and method call overhead
- Enable the use of intrinsics for readInt, readLong, etc.
- Availability of faster access methods like readFully and skipFully, without the need for helper functions
- Reduces some object creation in the performance critical path
This also includes and enables performance optimizations to:
- ByteBitPackingValuesReader
- PlainValuesReader
- RunLengthBitPackingHybridDecoder
Context:
I've been working on improving Parquet reading performance in Trino, mostly by profiling while running performance benchmarks and TPCDS queries. This PR is a subset of the changes I made that have more than doubled the performance of a lot of TPCDS queries (wall clock time, including the S3 access time). If you are kind enough to accept these changes, I have more I would like to contribute.
**Reporter**: [Timothy Miller](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=theosib-amazon) / @theosib-amazon
#### Externally tracked issue: [https://github.com/apache/parquet-mr/pull/953](https://github.com/apache/parquet-mr/pull/953)
**Note**: *This issue was originally created as [PARQUET-2135](https://issues.apache.org/jira/browse/PARQUET-2135). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
まず、外部で追跡されている pull request 953 と、指定されたコンポーネントである ByteBufferInputStream、LittleEndianDataInputStream、ByteBitPackingValuesReader、PlainValuesReader、RunLengthBitPackingHybridDecoder を確認します。関連する Trino と TPCDS のベンチマーク結果を比較します。マージされた機能と最適化がリグレッションなしで検証されれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java
- 領域
- data-engineering, performance
- issue の種類
- リファクタリング
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 25/100