Question on pig loader read parquet file
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
When I use spark save parquet file, schema like this
```
optional group attref (LIST) {
repeated group list {
optional group element {
optional binary nid (UTF8);
optional binary nss (UTF8);
}
}
}
```
And then use parquet-pig-bundle to read this file, the read function can work, but when i need to access "nid" it have some problem
If I read other file save by pig-storer, and need nid list, pig command is:
```
B = foreach A generate value.addr.clientIp_bag.clientIp, value.guid , value.attref.nid;
```
but read spark save version I need use this:
```
B = foreach M generate value.addr.clientIp, value.guid , flatten(value.attref);
C = foreach B generate clientIp, guid, attref::element.nid;
```
and this command will flatten column
My question is pig loader have some problem when loading parquet file(save by spark)
**Reporter**: [abel_ke](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=abel_ke)
**Note**: *This issue was originally created as [PARQUET-1172](https://issues.apache.org/jira/browse/PARQUET-1172). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
报告提到了 parquet-pig-bundle,并将 Spark 写入的 LIST schema 与 Pig-storer 的输出进行对比;先复现这两种读取方式,并跟踪 loader 如何暴露 attref 和 nid。完成标准是:Spark 生成的文件能够在不进行不必要的扁平化的情况下,支持预期的 nid 访问,并且涵盖所报告的 schema 和命令。
由索引模型根据 Issue 内容生成。
评估
- 领域
- data-engineering
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100