RecordReaderImplementation throw ArrayIndexOutOfBoundsException
- 主要言語
- Java
- スター
- 3.1k
- フォーク
- 1.6k
- 平均マージ
- 3日 12時間
- マージ済み PR(30日)
- 33
説明
Writing data by spark 3.1(parquet 1.10.1), sometimes it could get the error when reading it.
```java
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 141009 in block 1 in file hdfs://test/test_table/part-01423-58cbf8bc-5b0d-4d53-8279-2c8e3d293cc0.c000.zstd.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:251)
at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:177)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1509)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 254
at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:398)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
... 19 more
```
This table is a wide table(contains 160 columns), and it looks when it try to get a column definition level, which should be 0, but got 254 instead.
**Reporter**: [Hui An](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Bone+An)
**Note**: *This issue was originally created as [PARQUET-2298](https://issues.apache.org/jira/browse/PARQUET-2298). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
Start with RecordReaderImplementation.read at line 398 and the surrounding calls from InternalParquetRecordReader.nextKeyValue in the stack trace. Reproduce the failure using Spark 3.1, Parquet 1.10.1, and a wide table with about 160 columns, then inspect why a definition level of 254 is read where 0 is expected. Done means the affected file can be read without ArrayIndexOutOfBoundsException and the definition level is handled correctly.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java, spark
- 領域
- data-engineering
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 42/100