apache / apache/parquet-java

Page header uncompressed_page_size exceeds the actual decompressed payload by 2 bytes on ZSTD + dictionary-encoded ARRAY<ARRAY<BIGINT>> columns

Ouverte
#3,712 5 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Type: bug
Langage dominant
Java
Étoiles
3.1k
Forks
1.6k
Merge moyen
3 j 12 h
PR mergées (30 j)
33

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Summary

A corrupt Parquet file produced by a Spark writer (parquet-mr based, parquet 1.15.x, zstd-jni 1.5.2-1) contains data pages whose page header uncompressed_page_size is recorded 2 bytes larger than the bytes the decompressor actually produces. Reading
those pages throws EOFException at page decompression / level decoding time.

Environment

- Writer: Apache Spark (parquet-mr 1.15.2 vendored dependencies), ZSTD compression, dictionary encoding enabled (default).
- zstd-jni: 1.5.2-1 at write time (Spark). parquet 1.15.2 declares zstd-jni.version = 1.5.6-6.
- Reader: paimon vectorized reader (VectorizedColumnReader / VectorizedRleValuesReader) using parquet-mr 1.15.2, zstd-jni 1.5.7-6.
- Schema column: ARRAY> → leaf optional int64 element at path *.list.element.list.element, maxRepetitionLevel=2, maxDefinitionLevel=5, dictionary-encoded (PLAIN_DICTIONARY), V1 data pages (writer version PARQUET_1_0).

Error messages

Reading the affected page fails with:
org.apache.parquet.io.ParquetDecodingException: could not decompress page
at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:212)
...
Caused by: java.io.EOFException
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:399)
at org.apache.parquet.bytes.BytesInput.copy(BytesInput.java:205)
at org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:178)
at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:178)
After patching the header's uncompressed_page_size (3127 → 3125) so decompression succeeds, reading still fails, but inside level decoding:
java.io.EOFException
at org.apache.parquet.bytes.SingleBufferInputStream.sliceBuffers(SingleBufferInputStream.java:134)
at org.apache.parquet.bytes.ByteBufferInputStream.sliceStream(ByteBufferInputStream.java:116)
at org.apache.paimon.format.parquet.reader.VectorizedRleValuesReader.initFromPage(VectorizedRleValuesReader.java:108)
at org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:134)

Reproducer / measurements

On the corrupt file (178 MB, 6 row groups, 1431 leaf columns), scanning every page by independently decoding the thrift PageHeader and decompressing the page body with zstd-jni directly (bypassing the reader), only 2 pages are affected, both identical
in shape:
- row group 4, column 139 (55.list.element.list.element), data page 5
- row group 4, column 141 (57.list.element.list.element), data page 1

Both report:
header uncompressed_page_size = 3127
header compressed_page_size = 1877
valueCount = 2898
rlEnc=RLE dlEnc=RLE valEnc=PLAIN_DICTIONARY
zstd actual decompressed bytes = 3125 (delta = +2)

Decompressing with zstd-jni 1.5.2-1 and 1.5.7-6 both yield 3125, so this is not a zstd-jni version artifact — the writer fed 3125 bytes to zstd but recorded 3127 in the header.
When I decompressed according to 3125 and read the page content, I found that the declared data length for the RLE part was 658, but the actual content length was 656. It seems that the RLE part output 2 bytes less

### Component(s)

Core

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez par ColumnChunkPageReadStore et CodecFactory pour suivre la gestion de la taille des pages, puis examinez DataPageV1 et VectorizedRleValuesReader afin d’identifier l’écart de longueur RLE. Reproduisez les deux pages ZSTD concernées, encodées avec dictionnaire et contenant des tableaux imbriqués, décrites dans le rapport. Le travail est terminé lorsque ces pages sont décompressées et que le décodage des niveaux s’achève sans EOFException, en tenant compte de l’écart de 2 octets.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java, spark
Domaine
data-engineering
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Active
Clarté
Plutôt claire
Accessibilité débutants
48/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.