apache / apache/parquet-java

parquet-cli cannot display data containing int96 type

Abierto
#3,345 0 comentarios 1 reacción 0 asignados Ver en GitHub
Type: bug
Lenguaje dominante
Java
Estrellas
3.1k
Forks
1.6k
Merge medio
3 d 12 h
PR fusionados (30 d)
33

Descripción

### Describe the bug, including details regarding any error messages, version, and platform.

parquet-cli cannot display the contents of a Parquet file containing int96 types.
```shell
parquet cat part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
Argument error: INT96 is deprecated. As interim enable READ_INT96_AS_FIXED flag to read as byte array.
```
After adding the parameter `parquet.avro.readInt96AsFixed=true`, reading will still fail due to schema mismatch.
```shell
parquet -Dparquet.avro.readInt96AsFixed=true cat ~/Downloads/part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
Unknown error
java.lang.RuntimeException: Failed on record 0 in file /Users/admin/Downloads/part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
at org.apache.parquet.cli.commands.CatCommand.runWithAvroSchema(CatCommand.java:101)
at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:67)
at org.apache.parquet.cli.Main.run(Main.java:217)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.parquet.cli.Main.main(Main.java:245)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file file:/Users/admin/Downloads/part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:280)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:140)
at org.apache.parquet.cli.BaseCommand$2$1.advance(BaseCommand.java:505)
at org.apache.parquet.cli.BaseCommand$2$1.(BaseCommand.java:486)
at org.apache.parquet.cli.BaseCommand$2.iterator(BaseCommand.java:484)
at org.apache.parquet.cli.commands.CatCommand.runWithAvroSchema(CatCommand.java:88)
... 4 more
Caused by: org.apache.parquet.io.ParquetDecodingException: The requested schema is not compatible with the file schema. incompatible types: required fixed_len_byte_array(12) timestamp_col != required int96 timestamp_col
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.incompatibleSchema(ColumnIOFactory.java:104)
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visit(ColumnIOFactory.java:95)
at org.apache.parquet.schema.PrimitiveType.accept(PrimitiveType.java:693)
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visitChildren(ColumnIOFactory.java:83)
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visit(ColumnIOFactory.java:57)
at org.apache.parquet.schema.MessageType.accept(MessageType.java:52)
at org.apache.parquet.io.ColumnIOFactory.getColumnIO(ColumnIOFactory.java:167)
at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:155)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:245)
... 10 more
```

### Component(s)

CLI

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Reproduce the failure with parquet-cli cat on a Parquet file containing an INT96 column and review the stack trace through parquet.cli.commands.CatCommand.runWithAvroSchema and CatCommand.run. Trace how the file schema is converted for reading; done means parquet-cli can display records containing INT96 without the schema mismatch or decoding error.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
cli, data
Tipo de issue
Error
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
48/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.