apache / apache/parquet-java

parquet-cli cannot display data containing int96 type

Offen
#3,345 0 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
Type: bug
Vorherrschende Sprache
Java
Sterne
3.1k
Forks
1.6k
Ø Merge
3 T. 12 Std.
Gemergte PRs (30 T.)
33

Beschreibung

### Describe the bug, including details regarding any error messages, version, and platform.

parquet-cli cannot display the contents of a Parquet file containing int96 types.
```shell
parquet cat part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
Argument error: INT96 is deprecated. As interim enable READ_INT96_AS_FIXED flag to read as byte array.
```
After adding the parameter `parquet.avro.readInt96AsFixed=true`, reading will still fail due to schema mismatch.
```shell
parquet -Dparquet.avro.readInt96AsFixed=true cat ~/Downloads/part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
Unknown error
java.lang.RuntimeException: Failed on record 0 in file /Users/admin/Downloads/part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
at org.apache.parquet.cli.commands.CatCommand.runWithAvroSchema(CatCommand.java:101)
at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:67)
at org.apache.parquet.cli.Main.run(Main.java:217)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.parquet.cli.Main.main(Main.java:245)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file file:/Users/admin/Downloads/part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:280)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:140)
at org.apache.parquet.cli.BaseCommand$2$1.advance(BaseCommand.java:505)
at org.apache.parquet.cli.BaseCommand$2$1.(BaseCommand.java:486)
at org.apache.parquet.cli.BaseCommand$2.iterator(BaseCommand.java:484)
at org.apache.parquet.cli.commands.CatCommand.runWithAvroSchema(CatCommand.java:88)
... 4 more
Caused by: org.apache.parquet.io.ParquetDecodingException: The requested schema is not compatible with the file schema. incompatible types: required fixed_len_byte_array(12) timestamp_col != required int96 timestamp_col
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.incompatibleSchema(ColumnIOFactory.java:104)
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visit(ColumnIOFactory.java:95)
at org.apache.parquet.schema.PrimitiveType.accept(PrimitiveType.java:693)
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visitChildren(ColumnIOFactory.java:83)
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visit(ColumnIOFactory.java:57)
at org.apache.parquet.schema.MessageType.accept(MessageType.java:52)
at org.apache.parquet.io.ColumnIOFactory.getColumnIO(ColumnIOFactory.java:167)
at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:155)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:245)
... 10 more
```

### Component(s)

CLI

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Reproduce the failure with parquet-cli cat on a Parquet file containing an INT96 column and review the stack trace through parquet.cli.commands.CatCommand.runWithAvroSchema and CatCommand.run. Trace how the file schema is converted for reading; done means parquet-cli can display records containing INT96 without the schema mismatch or decoding error.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
cli, data
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
48/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.