Page header uncompressed_page_size exceeds the actual decompressed payload by 2 bytes on ZSTD + dictionary-encoded ARRAY<ARRAY<BIGINT>> columns
- 主要言語
- Java
- スター
- 3.1k
- フォーク
- 1.6k
- 平均マージ
- 3日 12時間
- マージ済み PR(30日)
- 33
説明
### Describe the bug, including details regarding any error messages, version, and platform.
Summary
A corrupt Parquet file produced by a Spark writer (parquet-mr based, parquet 1.15.x, zstd-jni 1.5.2-1) contains data pages whose page header uncompressed_page_size is recorded 2 bytes larger than the bytes the decompressor actually produces. Reading
those pages throws EOFException at page decompression / level decoding time.
Environment
- Writer: Apache Spark (parquet-mr 1.15.2 vendored dependencies), ZSTD compression, dictionary encoding enabled (default).
- zstd-jni: 1.5.2-1 at write time (Spark). parquet 1.15.2 declares zstd-jni.version = 1.5.6-6.
- Reader: paimon vectorized reader (VectorizedColumnReader / VectorizedRleValuesReader) using parquet-mr 1.15.2, zstd-jni 1.5.7-6.
- Schema column: ARRAY> → leaf optional int64 element at path *.list.element.list.element, maxRepetitionLevel=2, maxDefinitionLevel=5, dictionary-encoded (PLAIN_DICTIONARY), V1 data pages (writer version PARQUET_1_0).
Error messages
Reading the affected page fails with:
org.apache.parquet.io.ParquetDecodingException: could not decompress page
at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:212)
...
Caused by: java.io.EOFException
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:399)
at org.apache.parquet.bytes.BytesInput.copy(BytesInput.java:205)
at org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:178)
at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:178)
After patching the header's uncompressed_page_size (3127 → 3125) so decompression succeeds, reading still fails, but inside level decoding:
java.io.EOFException
at org.apache.parquet.bytes.SingleBufferInputStream.sliceBuffers(SingleBufferInputStream.java:134)
at org.apache.parquet.bytes.ByteBufferInputStream.sliceStream(ByteBufferInputStream.java:116)
at org.apache.paimon.format.parquet.reader.VectorizedRleValuesReader.initFromPage(VectorizedRleValuesReader.java:108)
at org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:134)
Reproducer / measurements
On the corrupt file (178 MB, 6 row groups, 1431 leaf columns), scanning every page by independently decoding the thrift PageHeader and decompressing the page body with zstd-jni directly (bypassing the reader), only 2 pages are affected, both identical
in shape:
- row group 4, column 139 (55.list.element.list.element), data page 5
- row group 4, column 141 (57.list.element.list.element), data page 1
Both report:
header uncompressed_page_size = 3127
header compressed_page_size = 1877
valueCount = 2898
rlEnc=RLE dlEnc=RLE valEnc=PLAIN_DICTIONARY
zstd actual decompressed bytes = 3125 (delta = +2)
Decompressing with zstd-jni 1.5.2-1 and 1.5.7-6 both yield 3125, so this is not a zstd-jni version artifact — the writer fed 3125 bytes to zstd but recorded 3127 in the header.
When I decompressed according to 3125 and read the page content, I found that the declared data length for the RLE part was 658, but the actual content length was 656. It seems that the RLE part output 2 bytes less
### Component(s)
Core
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
ColumnChunkPageReadStore と CodecFactory から始めてページサイズの処理を追跡し、次に DataPageV1 と VectorizedRleValuesReader を調べて RLE の長さの不一致を確認します。レポートで説明されている、影響を受ける 2 つの ZSTD の辞書エンコードされたネストされた配列ページを再現します。これらのページが展開され、2 バイトの差異を考慮したうえで EOFException なしにレベルのデコードを完了すれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java, spark
- 領域
- data-engineering
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 活発
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 48/100