Add differentiation of nested records with the same name
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
Hello,
While reading back a Parquet file produced with Spark, it appears the schema produced by Parquet-Avro is not valid.
I consider the simple following piece of code:
```Java
ParquetReader reader =
AvroParquetReader.builder(new org.apache.hadoop.fs.Path(path.toUri())).build();
System.out.println(reader.read().getSchema());
```
I get a stack lile:
```Java
Exception in thread "main" +org.apache.avro.SchemaParseException+: Can't redefine: value
at org.apache.avro.Schema$Names.put(+Schema.java:1128+)
at org.apache.avro.Schema$NamedSchema.writeNameRef(+Schema.java:562+)
at org.apache.avro.Schema$RecordSchema.toJson(+Schema.java:690+)
at org.apache.avro.Schema$UnionSchema.toJson(+Schema.java:882+)
at org.apache.avro.Schema$MapSchema.toJson(+Schema.java:833+)
at org.apache.avro.Schema$UnionSchema.toJson(+Schema.java:882+)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(+Schema.java:716+)
at org.apache.avro.Schema$RecordSchema.toJson(+Schema.java:701+)
at org.apache.avro.Schema.toString(+Schema.java:324+)
at org.apache.avro.Schema.toString(+Schema.java:314+)
```
The issue seems the same as the one reported in:
It have been fixed in Spark-avro within:
In our case, the parquet schema looks like:
```Java
message spark_schema {
optional group calculatedobjectinfomap (MAP) {
repeated group key_value {
required binary key (UTF8);
optional group value {
optional int64 calcobjid;
optional int64 calcobjparentid;
optional binary portfolioname (UTF8);
optional binary portfolioscheme (UTF8);
optional binary calcobjtype (UTF8);
optional binary calcobjmnemonic (UTF8);
optional binary calcobinstrumentype (UTF8);
optional int64 calcobjectqty;
optional binary calcobjboid (UTF8);
optional binary analyticalfoldermnemonic (UTF8);
optional binary calculatedidentifier (UTF8);
optional binary calcobjlevel (UTF8);
optional binary calcobjboidscheme (UTF8);
}
}
}
optional group riskfactorinfomap (MAP) {
repeated group key_value {
required binary key (UTF8);
optional group value {
optional binary riskfactorname (UTF8);
optional binary riskfactortype (UTF8);
optional binary riskfactorrole (UTF8);
}
}
}
}
```
We indeed have 2 Map field with a value fields named 'value'. The name 'value' is defaulted in org.apache.spark.sql.types.MapType.
The fix seems not trivial given current parquet-avro code then I doubt I will be able to craft a valid PR without directions.
Thanks,
**Reporter**: [Benoit Lacelle](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=blasd)
**Note**: *This issue was originally created as [PARQUET-1202](https://issues.apache.org/jira/browse/PARQUET-1202). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 AvroParquetReader.builder 以及读取所示 Parquet schema 时涉及的 schema 转换路径入手;检查 MapType 如何提供默认值名称。使用嵌套 map 示例重现该失败,并比较 spark-avro pull request 73 中引用的方法。当包含重复嵌套 record 名称的 schema 能够在不触发 Avro 的 "Can't redefine" 异常的情况下进行序列化时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100