apache / apache/parquet-java

Avro's isElementType() change breaks the reading of some parquet(1.8.1) files

Abierto
#2,381 13 comentarios 0 reacciones 0 asignados Ver en GitHub
Component: Avro Component: Parquet Priority: Critical Type: enhancement
Lenguaje dominante
Java
Estrellas
3.1k
Forks
1.6k
Merge medio
3 d 12 h
PR fusionados (30 d)
33

Descripción

When using the Avro schema below to write a parquet(1.8.1) file and then read back by using parquet 1.10.1 without passing any schema, the reading throws an exception "XXX is not a group" . Reading through parquet 1.8.1 is fine. 

           {

              "name": "phones",

              "type": [

                "null",

                {

                  "type": "array",

                  "items": {

                    "type": "record",

                    "name": "phones_items",

                    "fields": [

                      

{                         "name": "phone_number",                         "type": [                           "null",                           "string"                         ],                         "default": null                       }

                    ]

                  }

                }

              ],

              "default": null

            }

The code to read is as below 

     val reader = AvroParquetReader._builder_[SomeRecordType](parquetPath).withConf(**new**   Configuration).build()

    reader.read()

PARQUET-651 changed the method isElementType() by relying on Avro's checkReaderWriterCompatibility() to check the compatibility. However, checkReaderWriterCompatibility() consider the ParquetSchema and the AvroSchema(converted from File schema) as not compatible(the name in avro schema is ‘phones_items’, but the name is ‘array’ in Parquet schema, hence not compatible) . Hence return false and caused the “phone_number” field in the above schema to be considered as group type which is not true. Then the exception throws as .asGroupType(). 

I didn’t try writing via parquet 1.10.1 would reproduce the same problem or not. But it could because the translation of Avro schema to Parquet schema is not changed(didn’t verify yet). 

 I hesitate to revert PARQUET-651 because it solved several problems. I would like to hear the community's thoughts on it. 

**Reporter**: [Xinli Shang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shangx@uber.com) / @shangxinli
**Assignee**: [Xinli Shang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shangx@uber.com) / @shangxinli
#### Related issues:
- [Parquet-avro fails to decode array of record with a single field name "element" correctly](https://github.com/apache/parquet-java/issues/1976) (causes)

**Note**: *This issue was originally created as [PARQUET-1681](https://issues.apache.org/jira/browse/PARQUET-1681). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Reproduce el fallo con el esquema Avro proporcionado, la salida de parquet 1.8.1, parquet 1.10.1 y AvroParquetReader; después, inspecciona la lógica de isElementType() modificada por PARQUET-651 y su comprobación de compatibilidad. Se considera terminado cuando los archivos antiguos con este esquema de matriz de registros se puedan leer sin la excepción “not a group”, y el comportamiento esté cubierto por una prueba de regresión si el repositorio proporciona una.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
data-engineering
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.