apache / apache/parquet-java

parquet-cli cannot display data containing int96 type

オープン
#3,345 コメント 0 件 リアクション 1 件 担当者 0 名 GitHub で見る
Type: bug
主要言語
Java
スター
3.1k
フォーク
1.6k
平均マージ
3日 12時間
マージ済み PR(30日)
33

説明

### Describe the bug, including details regarding any error messages, version, and platform.

parquet-cli cannot display the contents of a Parquet file containing int96 types.
```shell
parquet cat part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
Argument error: INT96 is deprecated. As interim enable READ_INT96_AS_FIXED flag to read as byte array.
```
After adding the parameter `parquet.avro.readInt96AsFixed=true`, reading will still fail due to schema mismatch.
```shell
parquet -Dparquet.avro.readInt96AsFixed=true cat ~/Downloads/part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
Unknown error
java.lang.RuntimeException: Failed on record 0 in file /Users/admin/Downloads/part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
at org.apache.parquet.cli.commands.CatCommand.runWithAvroSchema(CatCommand.java:101)
at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:67)
at org.apache.parquet.cli.Main.run(Main.java:217)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.parquet.cli.Main.main(Main.java:245)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file file:/Users/admin/Downloads/part-00000-e4f30ffc-fdfa-465d-8205-23562b26616f-c000.zstd.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:280)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:140)
at org.apache.parquet.cli.BaseCommand$2$1.advance(BaseCommand.java:505)
at org.apache.parquet.cli.BaseCommand$2$1.(BaseCommand.java:486)
at org.apache.parquet.cli.BaseCommand$2.iterator(BaseCommand.java:484)
at org.apache.parquet.cli.commands.CatCommand.runWithAvroSchema(CatCommand.java:88)
... 4 more
Caused by: org.apache.parquet.io.ParquetDecodingException: The requested schema is not compatible with the file schema. incompatible types: required fixed_len_byte_array(12) timestamp_col != required int96 timestamp_col
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.incompatibleSchema(ColumnIOFactory.java:104)
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visit(ColumnIOFactory.java:95)
at org.apache.parquet.schema.PrimitiveType.accept(PrimitiveType.java:693)
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visitChildren(ColumnIOFactory.java:83)
at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visit(ColumnIOFactory.java:57)
at org.apache.parquet.schema.MessageType.accept(MessageType.java:52)
at org.apache.parquet.io.ColumnIOFactory.getColumnIO(ColumnIOFactory.java:167)
at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:155)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:245)
... 10 more
```

### Component(s)

CLI

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

Reproduce the failure with parquet-cli cat on a Parquet file containing an INT96 column and review the stack trace through parquet.cli.commands.CatCommand.runWithAvroSchema and CatCommand.run. Trace how the file schema is converted for reading; done means parquet-cli can display records containing INT96 without the schema mismatch or decoding error.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
cli, data
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
48/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。