NegativeArraySizeException on read for parquet files written with large strings in some cases
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
On Spark 3.3.1 which uses parquet 1.12.2, parquet files were successfully created using default parquet configs. Note: the write succeeded, so this is not the same as: https://issues.apache.org/jira/browse/PARQUET-1632
The payload had large strings and this resulted in the following exception on read:
```java
Caused by: java.lang.NegativeArraySizeException
at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:262)
at org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:214)
at org.apache.parquet.bytes.BytesInput.toInputStream(BytesInput.java:223)
at org.apache.parquet.column.impl.ColumnReaderImpl.readPageV1(ColumnReaderImpl.java:592)
at org.apache.parquet.column.impl.ColumnReaderImpl.access$300(ColumnReaderImpl.java:57)
at org.apache.parquet.column.impl.ColumnReaderImpl$3.visit(ColumnReaderImpl.java:536)
at org.apache.parquet.column.impl.ColumnReaderImpl$3.visit(ColumnReaderImpl.java:533)
at org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:95)
```
The issue could be addressed with the following configs:
```java
parquet.page.size.row.check.min=1
parquet.page.size.row.check.max=1000
parquet.page.size.check.estimate=false
spark.sql.parquet.columnarReaderBatchSize=2098
```
**Reporter**: [Atul Felix Payapilly](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=atulpayapilly_amazon)
**Note**: *This issue was originally created as [PARQUET-2367](https://issues.apache.org/jira/browse/PARQUET-2367). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
首先使用包含大型字符串的 Parquet 数据重现读取失败,并根据堆栈跟踪检查 BytesInput$StreamBytesInput.toByteArray、ColumnReaderImpl.readPageV1 和 DataPageV1.accept。将行为与列出的页面大小和 Spark reader 设置进行比较。当理解失败的读取并通过回归测试覆盖该问题,且不会出现 NegativeArraySizeException 时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 28/100