Negative initial size when writing large values in parquet-mr
- Lingua principale
- Java
- Stelle
- 3.1k
- Fork
- 1.6k
- Merge medio
- 3g 12h
- PR unite (30g)
- 33
Descrizione
I encountered an issue when writing large string values to Parquet.
Here is the code to reproduce the issue:
```java
import org.apache.spark.sql.functions._
def longString: String = "a" * (64 * 1024 * 1024)
val long_string = udf(() => longString)
val df = spark.range(0, 40, 1, 1).withColumn("large_str", long_string())
spark.conf.set("parquet.enable.dictionary", "false")
df.write.option("compression", "uncompressed").mode("overwrite").parquet("/tmp/large.parquet")
```
This Spark job fails with the exception:
```java
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 10861.0 failed 4 times, most recent failure: Lost task 0.3 in stage 10861.0 (TID 671168, 10.0.180.13, executor 14656): org.apache.spark.SparkException: Task failed while writing rows. at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257) at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170) at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.doRunTask(Task.scala:139) at org.apache.spark.scheduler.Task.run(Task.scala:112) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1526) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Negative initial size: -1610612543 at java.io.ByteArrayOutputStream.(ByteArrayOutputStream.java:74) at org.apache.parquet.bytes.BytesInput$BAOS.(BytesInput.java:234) at org.apache.parquet.bytes.BytesInput$BAOS.(BytesInput.java:232) at org.apache.parquet.bytes.BytesInput.toByteArray(BytesInput.java:202) at org.apache.parquet.bytes.ConcatenatingByteArrayCollector.collect(ConcatenatingByteArrayCollector.java:33) at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:126) at org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:147) at org.apache.parquet.column.impl.ColumnWriterV1.flush(ColumnWriterV1.java:235) at org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:122) at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:172) at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:114) at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165) at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42) at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57) at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74) at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247) at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1560) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248) ... 11 more
```
Would appreciate if you could help addressing the problem. Thanks!
**Reporter**: [Ivan Sadikov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sadikovi) / @sadikovi
**Assignee**: [Junjie Chen](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=junjie) / @chenjunjiedada
**Note**: *This issue was originally created as [PARQUET-1632](https://issues.apache.org/jira/browse/PARQUET-1632). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Inizia con il percorso dello stack trace attraverso BytesInput, ConcatenatingByteArrayCollector, ColumnChunkPageWriteStore e ColumnWriterV1. Esegui la riproduzione Spark fornita con valori stringa di grandi dimensioni e analizza il comportamento di scrittura delle pagine parquet. Il lavoro è completato quando la riproduzione non fallisce più con una dimensione negativa di ByteArrayOutputStream e questo caso è coperto da una regressione.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- java
- Ambito
- data-engineering
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 45/100