apache / apache/parquet-java

Parquet 1.9 not forward compatible with Parquet 1.10

未关闭
#2,589 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Parquet Priority: Major Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

This patch broke forward compatibility
https://issues.apache.org/jira/browse/PARQUET-357

This introduced a binary field which cannot be deserialized while reading from 1.9
New schema:
"type": { "id": "STRING", "binary": false }
Error message
```java

Caused by: java.lang.RuntimeException: shaded.parquet.org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "binary" (Class org.apache.parquet.thrift.struct.ThriftType$StringType), not marked as ignorable at [Source: java.io.StringReader@611b35d6; line: 23, column: 23] (through reference chain: org.apache.parquet.thrift.struct.StringType["binary"])
at org.apache.parquet.thrift.struct.JSON.fromJSON(JSON.java:38) at org.apache.parquet.thrift.struct.ThriftType.fromJSON(ThriftType.java:89) at org.apache.parquet.thrift.ThriftMetaData.parseDescriptor(ThriftMetaData.java:121)
... 19 more Caused by: shaded.parquet.org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "binary" (Class org.apache.parquet.thrift.struct.ThriftType$StringType), not marked as ignorable
```
A simple fix to 1.9 should fix this

 
```java

--- a/parquet-thrift/src/main/java/org/apache/parquet/thrift/struct/ThriftType.java
+++ b/parquet-thrift/src/main/java/org/apache/parquet/thrift/struct/ThriftType.java
@@ -641,11 +641,23 @@ public abstract class ThriftType {
}
public static class StringType extends ThriftType {
+ private boolean binary = false;
+
+ @JsonIgnore
+ public boolean isBinary() {
+ return binary;
+ }
+
+ @JsonProperty
+ public void setBinary(boolean binary) {
+ this.binary = binary;
+ }
```

**Reporter**: [Nikhil Goyal](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=nownikhil)

**Note**: *This issue was originally created as [PARQUET-2002](https://issues.apache.org/jira/browse/PARQUET-2002). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 parquet-thrift/src/main/java/org/apache/parquet/thrift/struct/ThriftType.java 开始,重点关注 StringType 以及 stack trace 中显示的 JSON 解析路径。使用 1.9 reader 重现读取包含 binary 字段的 schema,并将其与使用 1.10 生成的 schema 进行比较。完成标准是旧 reader 能够在不出现 UnrecognizedPropertyException 的情况下接受新 schema。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。