apache / apache/parquet-java

Incompatible behavior for ColumnChunk.file_offset between Parquet-mr and Impala

Aperta
#1,516 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Component: Format Component: Java Component: Parquet Priority: Major Type: bug
Lingua principale
Java
Stelle
3.1k
Fork
1.6k
Merge medio
3g 12h
PR unite (30g)
33

Descrizione

According to comments in [parquet.thrift](https://github.com/apache/incubator-parquet-format/blob/master/src/thrift/parquet.thrift#L479), this field is supposed to store offset of ColumnMetaData within the file column chunk is stored. My understanding is that this allows omitting ColumnMetaData within ColumnChunk (it is optional field, after all). Unfortunately, two major implementations, Parquet-mr and Impala, deviate from this definition when writing Parquet files. Impala implementation writes offset pointing to the ColumnChunk rather than ColumnMetaData, as can be found in [hdfs-parquet-table-reader.cc](https://github.com/cloudera/Impala/blob/24db37f4efdc493d218470dc045b61f5104c4fd0/be/src/exec/hdfs-parquet-table-writer.cc#L895). While this is still incorrect behavior according to the comments in parquet.thrift, this still allows access to the ColumnMetaData necessary for reading data.

Parquet-mr implementation can be found in [ParquetMetadataConverter](https://github.com/Parquet/parquet-mr/blob/fd8d18f26af9ad7813dda71352b5dcb0080306eb/parquet-hadoop/src/main/java/parquet/format/converter/ParquetMetadataConverter.java#L149), which writes the offset to the first data page. Not only this is incompatible behavior, but also it makes no sense because you cannot read the data with just data page offset. There is even a comment on that line saying "verify this is the right offset."

**Reporter**: [Eunsoo Roh](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=eunsoor)

**Note**: *This issue was originally created as [PARQUET-68](https://issues.apache.org/jira/browse/PARQUET-68). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Leggi src/thrift/parquet.thrift e ParquetMetadataConverter.java, quindi confronta il ColumnChunk.file_offset documentato con il comportamento del writer Impala a cui si fa riferimento. Determina il contratto di offset compatibile tra queste implementazioni; il lavoro richiede di risolvere la discrepanza tra la specifica e l’implementazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
data-engineering
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.