RecordReaderImplementation throw ArrayIndexOutOfBoundsException
- Langage dominant
- Java
- Étoiles
- 3.1k
- Forks
- 1.6k
- Merge moyen
- 3 j 12 h
- PR mergées (30 j)
- 33
Description
Writing data by spark 3.1(parquet 1.10.1), sometimes it could get the error when reading it.
```java
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 141009 in block 1 in file hdfs://test/test_table/part-01423-58cbf8bc-5b0d-4d53-8279-2c8e3d293cc0.c000.zstd.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:251)
at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:177)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1509)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 254
at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:398)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
... 19 more
```
This table is a wide table(contains 160 columns), and it looks when it try to get a column definition level, which should be 0, but got 254 instead.
**Reporter**: [Hui An](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Bone+An)
**Note**: *This issue was originally created as [PARQUET-2298](https://issues.apache.org/jira/browse/PARQUET-2298). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Start with RecordReaderImplementation.read at line 398 and the surrounding calls from InternalParquetRecordReader.nextKeyValue in the stack trace. Reproduce the failure using Spark 3.1, Parquet 1.10.1, and a wide table with about 160 columns, then inspect why a definition level of 254 is read where 0 is expected. Done means the affected file can be read without ArrayIndexOutOfBoundsException and the definition level is handled correctly.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java, spark
- Domaine
- data-engineering
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 42/100