For array type, inconsistent names are used as the array element name.
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
When creating a convert for an array, Parquet Avro uses "array" as the field name name ([see here](https://github.com/apache/incubator-parquet-mr/blob/parquet-1.6.0rc7/parquet-avro/src/main/java/parquet/avro/AvroSchemaConverter.java#L131)) , but Parquet Hive SerDe uses "array_element" as the field name [see here](https://github.com/apache/incubator-parquet-mr/blob/parquet-1.6.0rc7/parquet-hive/parquet-hive-storage-handler/src/main/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java#L109). In Spark SQL, our native Parquet support is following Parquet Avro's convention, for data generated by Parquet Hive SerDe, the array value cannot be correctly read and null will be returned.
**Reporter**: [Yin Huai](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=yhuai)
#### Related issues:
- [Arrays and Maps stored with Hive Parquet Serde may not be able to read by the Parquet support in the Data Souce API](https://issues.apache.org/jira/browse/SPARK-5508) (is related to)
**Note**: *This issue was originally created as [PARQUET-221](https://issues.apache.org/jira/browse/PARQUET-221). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
比较 issue 中链接的 parquet-avro 的 AvroSchemaConverter.java 和 parquet-hive 的 HiveSchemaConverter.java 中数组元素的命名方式。通过 Spark SQL 的原生 Parquet 支持复现读取 Hive Parquet 数组数据的过程,然后验证数组不再返回 null,同时保持与现有文件的兼容性。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100