apache / apache/parquet-java

Avro's isElementType() change breaks the reading of some parquet(1.8.1) files

Ouverte
#2,381 13 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Component: Avro Component: Parquet Priority: Critical Type: enhancement
Langage dominant
Java
Étoiles
3.1k
Forks
1.6k
Merge moyen
3 j 12 h
PR mergées (30 j)
33

Description

When using the Avro schema below to write a parquet(1.8.1) file and then read back by using parquet 1.10.1 without passing any schema, the reading throws an exception "XXX is not a group" . Reading through parquet 1.8.1 is fine. 

           {

              "name": "phones",

              "type": [

                "null",

                {

                  "type": "array",

                  "items": {

                    "type": "record",

                    "name": "phones_items",

                    "fields": [

                      

{                         "name": "phone_number",                         "type": [                           "null",                           "string"                         ],                         "default": null                       }

                    ]

                  }

                }

              ],

              "default": null

            }

The code to read is as below 

     val reader = AvroParquetReader._builder_[SomeRecordType](parquetPath).withConf(**new**   Configuration).build()

    reader.read()

PARQUET-651 changed the method isElementType() by relying on Avro's checkReaderWriterCompatibility() to check the compatibility. However, checkReaderWriterCompatibility() consider the ParquetSchema and the AvroSchema(converted from File schema) as not compatible(the name in avro schema is ‘phones_items’, but the name is ‘array’ in Parquet schema, hence not compatible) . Hence return false and caused the “phone_number” field in the above schema to be considered as group type which is not true. Then the exception throws as .asGroupType(). 

I didn’t try writing via parquet 1.10.1 would reproduce the same problem or not. But it could because the translation of Avro schema to Parquet schema is not changed(didn’t verify yet). 

 I hesitate to revert PARQUET-651 because it solved several problems. I would like to hear the community's thoughts on it. 

**Reporter**: [Xinli Shang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shangx@uber.com) / @shangxinli
**Assignee**: [Xinli Shang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shangx@uber.com) / @shangxinli
#### Related issues:
- [Parquet-avro fails to decode array of record with a single field name "element" correctly](https://github.com/apache/parquet-java/issues/1976) (causes)

**Note**: *This issue was originally created as [PARQUET-1681](https://issues.apache.org/jira/browse/PARQUET-1681). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Reproduisez l’échec avec le schéma Avro fourni, la sortie de parquet 1.8.1, parquet 1.10.1 et AvroParquetReader, puis examinez la logique isElementType() modifiée par PARQUET-651 ainsi que sa vérification de compatibilité. La tâche est terminée lorsque les anciens fichiers utilisant ce schéma de tableau d’enregistrements peuvent être lus sans l’exception « not a group », et que le comportement est couvert par un test de régression si le dépôt en fournit un.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java
Domaine
data-engineering
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.