Corrupt Parquet Files

I am getting corrupt parquet files as the result of a spark job. The write job completes with no errors but when I read the data again I get the following error:

org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file hdfs://MYPATH/part-r-00004-b5c93a19-2f75-4c04-b798-de9cb463f02f.gz.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228)
at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:91)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:128)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:91)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithoutKey$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NegativeArraySizeException
at org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:755)
at org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:494)
at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:208)

The job that generates this data partitions and sorts the data in a particular way to achieve better compression. If I don't partition and sort I have not been able to reproduce its behavior. It also only has this behavior on say 25% of the data. Most of the time simply rerunning the write job would cause the read error to go away but I have now run across cases where that was not the case. I am happy to give what data I can, or work with someone to run this down.

I know this is a sub-optimal report, but I have not been able to randomly generate data to reproduce this issue. The data that trips this bug is typically 5GB+ post compression files.

**Environment**: HDP-2.5.3.0 Spark-2.0.2
**Reporter**: [Steve Severance](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sseveran)

_{**Note**: *This issue was originally created as [PARQUET-831](https://issues.apache.org/jira/browse/PARQUET-831). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

Commencez par les points d’entrée de la stack trace ParquetFileReader.ConsecutiveChunkList.readAll et InternalParquetRecordReader, puis étudiez les écritures impliquant le partitionnement et le tri de Spark. Utilisez l’environnement signalé HDP-2.5.3.0 et Spark 2.0.2 ainsi que de gros fichiers compressés afin de rechercher un cas reproductible. Le travail est considéré comme terminé lorsque la cause est identifiée et qu’il est vérifié que les fichiers Parquet générés peuvent être lus sans l’exception signalée.

Rédigé par le modèle d'indexation à partir du texte de l'issue.