Avro's isElementType() change breaks the reading of some parquet(1.8.1) files
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
When using the Avro schema below to write a parquet(1.8.1) file and then read back by using parquet 1.10.1 without passing any schema, the reading throws an exception "XXX is not a group" . Reading through parquet 1.8.1 is fine.
{
"name": "phones",
"type": [
"null",
{
"type": "array",
"items": {
"type": "record",
"name": "phones_items",
"fields": [
{ "name": "phone_number", "type": [ "null", "string" ], "default": null }
]
}
}
],
"default": null
}
The code to read is as below
val reader = AvroParquetReader._builder_[SomeRecordType](parquetPath).withConf(**new** Configuration).build()
reader.read()
PARQUET-651 changed the method isElementType() by relying on Avro's checkReaderWriterCompatibility() to check the compatibility. However, checkReaderWriterCompatibility() consider the ParquetSchema and the AvroSchema(converted from File schema) as not compatible(the name in avro schema is ‘phones_items’, but the name is ‘array’ in Parquet schema, hence not compatible) . Hence return false and caused the “phone_number” field in the above schema to be considered as group type which is not true. Then the exception throws as .asGroupType().
I didn’t try writing via parquet 1.10.1 would reproduce the same problem or not. But it could because the translation of Avro schema to Parquet schema is not changed(didn’t verify yet).
I hesitate to revert PARQUET-651 because it solved several problems. I would like to hear the community's thoughts on it.
**Reporter**: [Xinli Shang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shangx@uber.com) / @shangxinli
**Assignee**: [Xinli Shang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shangx@uber.com) / @shangxinli
#### Related issues:
- [Parquet-avro fails to decode array of record with a single field name "element" correctly](https://github.com/apache/parquet-java/issues/1976) (causes)
**Note**: *This issue was originally created as [PARQUET-1681](https://issues.apache.org/jira/browse/PARQUET-1681). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
使用提供的 Avro schema、parquet 1.8.1 输出、parquet 1.10.1 和 AvroParquetReader 重现该失败,然后检查由 PARQUET-651 修改的 isElementType() 逻辑及其兼容性检查。完成的标准是:使用此数组记录 schema 的旧文件可以读取且不会出现“not a group”异常;如果 repository 提供 regression test,则应通过 regression test 覆盖该行为。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100