apache / apache/parquet-java

Schema mismatch for reading Avro from parquet file with old schema version?

Ouverte
#1,628 5 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Component: Avro Component: Parquet Priority: Minor Type: bug
Langage dominant
Java
Étoiles
3.1k
Forks
1.6k
Merge moyen
3 j 12 h
PR mergées (30 j)
33

Description

I ran into what looks like a bug in the Parquet Avro reading code, around trying to read a file written with a previous version of a schema with a new, evolved version of the schema.

I'm using Apache Beam's ParquetIO library, which supports passing in schemas to use for "projection" and I was investigating if that would work for me here. However, it didn't work, complaining that my new reader schema had a field that wasn't in the writer schema.

 

I traced this through to a couple places in the parquet-avro code that don't look right to me:

 

First, in `prepareForRead` here:

The `parquetSchema` var comes from `parquetSchema = readContext.getRequestedSchema();` while the `avroSchema` var comes from the parquet file itself with `avroSchema = new Schema.Parser().parse(keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY));`

I can verify that `parquetSchema` is the schema I'm requesting it be projected to and that `avroSchema` is the schema from the file, but the naming looks backward, shouldn't `parquetSchema` be the one from the parquet file?

Following the stack down, I was hitting this line: https://github.com/apache/parquet-mr/blob/master/parquet-avro/src/main/java/org/apache/parquet/avro/AvroIndexedRecordConverter.java#L91

here it was failing because the `avroSchema` didn't have a field that was in the `parquetSchema`, with the variables assigned in the same way as above. That's the case I was hoping to use this projection for, though - to get the record read with the new reader schema, using the default value from the new schema for the new field. In fact, the comment on line 101 "store defaults for any new Avro fields from avroSchema that are not in the writer schema (parquetSchema)" suggests that the intent was for this to work, but the actual code has the writer schema in avroSchema and the reader schema in parquetSchema.

(Additionally, I'd want this to support schema evolution both for adding an optional field and also removing an old field - so just flipping the names around would result in this still breaking if the reader schema dropped a field from the writer schema...)

Looking to understand if I'm interpreting this correctly, or if there's another path that's intended to be used.

Thank you!

**Environment**: Linux, Apache Beam 2.28.0, Java 11
**Reporter**: [Philip Wilcox](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=philipwilcox)

**Note**: *This issue was originally created as [PARQUET-2055](https://issues.apache.org/jira/browse/PARQUET-2055). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez dans parquet-avro/src/main/java/org/apache/parquet/avro/AvroReadSupport.java, au niveau de prepareForRead, et suivez les schémas jusqu’à AvroIndexedRecordConverter.java, vers la ligne 91. Comparez le schéma de projection demandé au schéma de métadonnées Avro du fichier, y compris les commentaires sur les valeurs par défaut. Le travail est terminé lorsque le comportement attendu de l’évolution du schéma pour les champs ajoutés et supprimés est clarifié ou corrigé, avec une validation par rapport au scénario Apache Beam signalé.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java
Domaine
data-engineering
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.