apache / apache/parquet-java

Parquet reader throws error "Reading past RLE/BitPacking stream" for parquet file with null values

未关闭
#1,555 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Java Component: Parquet Priority: Major Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

Recently moved from parquet 1.8.x to 1.12 recently.

Dataset has > 20k null values to be written to a complex type. Earlier with 1.8.x, it would create single page but with 1.12 it creates 20 pages (parquet - 1414). Writing nulls to complex types has been optimised to be cached (null cache) that would be flushed on next non null encounter or explicit flush/close. With 1.8, it would have encountered explicit close and flush the null cache and write the page. But with 1.12, after encountering 20k values, the page is written prematurely.

 

Below is the metadata dump in both cases.

1.8 :

index._id TV=111396 RL=0 DL=2 ---------------------------------------------------------------------------- page 0: DLE:RLE RLE:BIT_PACKED VLE:PLAIN ST:[num_nulls: 111396, min/max not defined] SZ:8 VC:111396

 

1.12 :

index._index TV=111396 RL=0 DL=2 ---------------------------------------------------------------------------- page 0: DLE:RLE RLE:BIT_PACKED VLE:PLAIN ST:[no stats for this column] SZ:4 VC:0 ...... page 19: DLE:RLE RLE:BIT_PACKED VLE:PLAIN ST:[no stats for this column] SZ:8 VC:111396

All the pages in 1.12 except the last page have same metadata. Now the issue is when the parquet reader kicks in, it sees that the RLE is bit packed and reads 8 bytes which goes beyond the stream as the size is only 4 (Reading past RLE/BitPacking stream).

**Reporter**: [shyam narayan singh](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shyamsingh) / @shyambits2004

**Note**: *This issue was originally created as [PARQUET-1575](https://issues.apache.org/jira/browse/PARQUET-1575). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先复现报告的 Parquet 1.12 案例:复杂类型中包含超过 20k 个 null 值;然后根据提供的页面元数据检查 reader 对 RLE/bit-packed stream 的处理。完成标准是生成的页面可以读取且不会出现 "Reading past RLE/BitPacking stream" 错误,并且 1.8 与 1.12 的行为得到统一。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。