Parquet 1.9 not forward compatible with Parquet 1.10
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
This patch broke forward compatibility
https://issues.apache.org/jira/browse/PARQUET-357
This introduced a binary field which cannot be deserialized while reading from 1.9
New schema:
"type": { "id": "STRING", "binary": false }
Error message
```java
Caused by: java.lang.RuntimeException: shaded.parquet.org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "binary" (Class org.apache.parquet.thrift.struct.ThriftType$StringType), not marked as ignorable at [Source: java.io.StringReader@611b35d6; line: 23, column: 23] (through reference chain: org.apache.parquet.thrift.struct.StringType["binary"])
at org.apache.parquet.thrift.struct.JSON.fromJSON(JSON.java:38) at org.apache.parquet.thrift.struct.ThriftType.fromJSON(ThriftType.java:89) at org.apache.parquet.thrift.ThriftMetaData.parseDescriptor(ThriftMetaData.java:121)
... 19 more Caused by: shaded.parquet.org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "binary" (Class org.apache.parquet.thrift.struct.ThriftType$StringType), not marked as ignorable
```
A simple fix to 1.9 should fix this
```java
--- a/parquet-thrift/src/main/java/org/apache/parquet/thrift/struct/ThriftType.java
+++ b/parquet-thrift/src/main/java/org/apache/parquet/thrift/struct/ThriftType.java
@@ -641,11 +641,23 @@ public abstract class ThriftType {
}
public static class StringType extends ThriftType {
+ private boolean binary = false;
+
+ @JsonIgnore
+ public boolean isBinary() {
+ return binary;
+ }
+
+ @JsonProperty
+ public void setBinary(boolean binary) {
+ this.binary = binary;
+ }
```
**Reporter**: [Nikhil Goyal](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=nownikhil)
**Note**: *This issue was originally created as [PARQUET-2002](https://issues.apache.org/jira/browse/PARQUET-2002). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 parquet-thrift/src/main/java/org/apache/parquet/thrift/struct/ThriftType.java 开始,重点关注 StringType 以及 stack trace 中显示的 JSON 解析路径。使用 1.9 reader 重现读取包含 binary 字段的 schema,并将其与使用 1.10 生成的 schema 进行比较。完成标准是旧 reader 能够在不出现 UnrecognizedPropertyException 的情况下接受新 schema。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data
- Issue 类型
- 缺陷
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 活跃度
- 停滞
- 描述清晰度
- 描述清楚
- 新手友好度
- 35/100