ProtoReader does not iterate over the parquet file correctly.
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
The `ProtoParquetReader` does not iterate over the parquet file correctly, but it gets stuck in the first element and keeps reading as many times as elements the file contained.
In my Scala example I am just reading from a local file that I know for sure it contains right data.
```
val hadoopCOnf = new Configuration()
val outfile: String = genTemporaryFile()
val r: ParquetReader[Event.Builder] = {
ProtoParquetReader.builder[Event.Builder](new Path(outfile)).withConf(hadoopCOnf).build()
}
```
Notice that the proto schema that I am using is generated from
The generated proto implements com.google.protobuf.GeneratedMessageV3.
See an example on how the ProtoParquetReader is created line(65 and 69):
and here how it is used (notice that is defined for only one record, the same one for multiple records would fail)
**Reporter**: [Pau Alarcon ](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=paualarco)
**Note**: *This issue was originally created as [PARQUET-1871](https://issues.apache.org/jira/browse/PARQUET-1871). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 ProtoParquetReader 开始,比较它在 monix-connect 的 ProtoParquetFixture.scala 第 65-69 行附近的构造方式与在 ProtoParquetSpec.scala 第 83 行附近的使用方式。使用多条记录复现该行为,并验证迭代返回的是后续记录,而不是重复第一条记录。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java, scala
- 领域
- data-engineering
- Issue 类型
- 缺陷
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100