Page header uncompressed_page_size exceeds the actual decompressed payload by 2 bytes on ZSTD + dictionary-encoded ARRAY<ARRAY<BIGINT>> columns
- Vorherrschende Sprache
- Java
- Sterne
- 3.1k
- Forks
- 1.6k
- Ø Merge
- 3 T. 12 Std.
- Gemergte PRs (30 T.)
- 33
Beschreibung
### Describe the bug, including details regarding any error messages, version, and platform.
Summary
A corrupt Parquet file produced by a Spark writer (parquet-mr based, parquet 1.15.x, zstd-jni 1.5.2-1) contains data pages whose page header uncompressed_page_size is recorded 2 bytes larger than the bytes the decompressor actually produces. Reading
those pages throws EOFException at page decompression / level decoding time.
Environment
- Writer: Apache Spark (parquet-mr 1.15.2 vendored dependencies), ZSTD compression, dictionary encoding enabled (default).
- zstd-jni: 1.5.2-1 at write time (Spark). parquet 1.15.2 declares zstd-jni.version = 1.5.6-6.
- Reader: paimon vectorized reader (VectorizedColumnReader / VectorizedRleValuesReader) using parquet-mr 1.15.2, zstd-jni 1.5.7-6.
- Schema column: ARRAY> → leaf optional int64 element at path *.list.element.list.element, maxRepetitionLevel=2, maxDefinitionLevel=5, dictionary-encoded (PLAIN_DICTIONARY), V1 data pages (writer version PARQUET_1_0).
Error messages
Reading the affected page fails with:
org.apache.parquet.io.ParquetDecodingException: could not decompress page
at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:212)
...
Caused by: java.io.EOFException
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:399)
at org.apache.parquet.bytes.BytesInput.copy(BytesInput.java:205)
at org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:178)
at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:178)
After patching the header's uncompressed_page_size (3127 → 3125) so decompression succeeds, reading still fails, but inside level decoding:
java.io.EOFException
at org.apache.parquet.bytes.SingleBufferInputStream.sliceBuffers(SingleBufferInputStream.java:134)
at org.apache.parquet.bytes.ByteBufferInputStream.sliceStream(ByteBufferInputStream.java:116)
at org.apache.paimon.format.parquet.reader.VectorizedRleValuesReader.initFromPage(VectorizedRleValuesReader.java:108)
at org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:134)
Reproducer / measurements
On the corrupt file (178 MB, 6 row groups, 1431 leaf columns), scanning every page by independently decoding the thrift PageHeader and decompressing the page body with zstd-jni directly (bypassing the reader), only 2 pages are affected, both identical
in shape:
- row group 4, column 139 (55.list.element.list.element), data page 5
- row group 4, column 141 (57.list.element.list.element), data page 1
Both report:
header uncompressed_page_size = 3127
header compressed_page_size = 1877
valueCount = 2898
rlEnc=RLE dlEnc=RLE valEnc=PLAIN_DICTIONARY
zstd actual decompressed bytes = 3125 (delta = +2)
Decompressing with zstd-jni 1.5.2-1 and 1.5.7-6 both yield 3125, so this is not a zstd-jni version artifact — the writer fed 3125 bytes to zstd but recorded 3127 in the header.
When I decompressed according to 3125 and read the page content, I found that the declared data length for the RLE part was 658, but the actual content length was 656. It seems that the RLE part output 2 bytes less
### Component(s)
Core
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Rechercherichtung
Beginnen Sie mit ColumnChunkPageReadStore und CodecFactory, um die Behandlung der Seitengröße nachzuverfolgen, und untersuchen Sie anschließend DataPageV1 und VectorizedRleValuesReader auf die RLE-Längenabweichung. Reproduzieren Sie die beiden betroffenen, im Bericht beschriebenen ZSTD-Seiten mit Dictionary-Kodierung und verschachtelten Arrays. Als abgeschlossen gilt die Aufgabe, wenn diese Seiten dekomprimiert werden und die Level-Decodierung ohne EOFException abgeschlossen wird, wobei die Abweichung von 2 Byte berücksichtigt ist.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java, spark
- Bereich
- data-engineering
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Aktiv
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 48/100