apache / apache/parquet-java

AvroReadSupport.setRequestedProjection in 1.11.1+ not backwards compatible with MAPS

未关闭
#2,610 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Avro Component: Java Component: Parquet Priority: Minor Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

 

using AvroReadSupport.setRequestedProjection in 1.11.1 reading a 1.10.0 generated parquet file sets MAP fields to null (and vice versa 1.10.0 reader against a 1.11.1 generated file) 

- Not using a projected schema works, the map fields are converted correctly.
In my case. Parquet file is generated by hive 
```java

CREATE TABLE parquetmaptest (
a string,
b MAP
)
STORED AS PARQUET
tblproperties(
"parquet.compression"="SNAPPY"
);
insert into parquetmaptest select "a",map("k1","v1","k2","v2");
```

Using parquet-avro 1.11.1 (and appropriate dependencies) result in field "b" being null. 
```java

data:null
row:{"a": "a", "b": null}
```
Using parquet-avro 1.11.0 (and appropriate dependencies) result in field "b" being  the right map value.  
```java

data:{k1=v1, k2=v2}
row:{"a": "a", "b": {"k1": "v1", "k2": "v2"}}
```
 

**Reporter**: [Bil Bingham](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bilbingham)
#### Original Issue Attachments:
- [PQMapTest.java](https://issues.apache.org/jira/secure/attachment/13024749/PQMapTest.java)

**Note**: *This issue was originally created as [PARQUET-2039](https://issues.apache.org/jira/browse/PARQUET-2039). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

从附带的 PQMapTest.java 和 AvroReadSupport.setRequestedProjection 路径开始,在 Hive 生成的 MAP 文件上比较 parquet-avro 1.11.1 和 1.10.0。先复现 projected-read 失败;当文件和 reader 使用任一版本时,投影的 map 字段仍保持有值,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。