apache / apache/parquet-java

Inconsistency between vectorized processing boundary calculation in benchmark and readBatchUsing512Vector calculation method in ParquetReadRouter class?

オープン
#3,073 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Java
スター
3.1k
フォーク
1.6k
平均マージ
3日 12時間
マージ済み PR(30日)
33

説明

Hello, I have recently become interested in using vector API Parquet Bit-Packing decode. In the course of researching the code, I found that in the ByteBitPackingVectorBenchmarks.java in the official test benchmarks in the parquet-plugins-benchmarks folder totalByteCountVector = totalBytesCount - inputByteCountPerVector; When this range is exceeded, unpack8Values() is used to decode the data, which ensures that there is enough room for a full vector operation at the end. But in readBatchUsing512Vector() totalByteCountVector = totalBytesCount - BYTES_PER_VECTOR_512;
I am wondering if this affects the throughput performance and logic implementation choices for different bit width decoding methods.
My queries are as follows:
1. In this case, if it is the same amount of data, the number of vectorized decoding will be reduced, won't this affect the optimization effect after combining with Spark?
For example, bitwidth = 3, outputValues = 2048; when ByteBitPackingVectorBenchmarks.java totalByteCountVector = totalBytesCount - inputByteCountPerVector. Here the decoding is done by 63 times unpackValuesUsingVector and 4 times unpack8value. When totalByteCountVector = totalBytesCount - BYTES_PER_VECTOR_512; here the decoding is done by 59 times unpackValuesUsingVector, 20 times unpack8value.
![totalVectorCount](https://github.com/user-attachments/assets/6a80b473-e963-4543-9b20-6abf0af2f320)

2. I would like to ask here to reserve 64 bytes is to take into account the data out of bounds and other data security considerations?
Is it possible to use totalByteCountVector = totalBytesCount - inputByteCountPerVector; in the readBatchUsing512Vector boundary calculation code?
3. Also if readBatchUsing512Vector keeps totalByteCountVector = totalBytesCount - BYTES_PER_VECTOR_512; the vectorization bounds here are enough to cover off the bounds-safe case, Loads a vector from an array of type byte[] starting at an offset and using a mask is necessary?
![load Vector Using Mask or not](https://github.com/user-attachments/assets/a4c7308b-f829-4bf3-8acd-dcfdcb56c710)
As I understand it, boundary crossing for vectorized data processing would be mitigated if BYTES_PER_VECTOR_512 is used. The performance loss caused by using masks for boundary data reading safety is greater than static ByteVector fromArray(VectorSpecies species, byte[] a, int offset) which doesn't use masks but directly loads the excess part of the array unused. Would it be possible to consider loading a vector only from an array of type byte[] starting at offset, without using a mask? Although this would add some single decode extra data to the vector loading process? Or is there some other reason why I can't eliminate the mask, I'm just eagerly waiting for an answer.Sorry for my poor English.

### Component(s)

Core, Benchmark

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

ByteBitPackingVectorBenchmarks.java と ParquetReadRouter.readBatchUsing512Vector() の境界計算を比較し、指定されたビット幅と出力サイズに対するベクトル化処理と unpack8Values() 処理に注目してください。各選択肢のベンチマークを実行し、マスク付きおよびマスクなしのベクトルロードを調査して、境界と安全性に関する挙動が正しいかどうかを判断してください。結果によって実装の選択と必要な変更が裏付けられれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
data-engineering, performance
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
38/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。