apache / apache/parquet-java

ProtoReader does not iterate over the parquet file correctly.

Open
#2,497 0 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

The `ProtoParquetReader` does not iterate over the parquet file correctly, but it gets stuck in the first element and keeps reading as many times as elements the file contained.

In my Scala example I am just reading from a local file that I know for sure it contains right data.

```

val hadoopCOnf = new Configuration()

val outfile: String = genTemporaryFile()

val r: ParquetReader[Event.Builder] = {
ProtoParquetReader.builder[Event.Builder](new Path(outfile)).withConf(hadoopCOnf).build()
}

```

Notice that the proto schema that I am using is generated from 

The generated proto implements com.google.protobuf.GeneratedMessageV3.

See an example on how the ProtoParquetReader is created line(65 and 69): 

and here how it is used (notice that is defined for only one record, the same one for multiple records would fail) 

 

**Reporter**: [Pau Alarcon ](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=paualarco)

**Note**: *This issue was originally created as [PARQUET-1871](https://issues.apache.org/jira/browse/PARQUET-1871). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ProtoParquetReader and compare its construction in monix-connect's ProtoParquetFixture.scala around lines 65-69 with its use in ProtoParquetSpec.scala around line 83. Reproduce the behavior with multiple records and verify that iteration returns successive records rather than repeating the first one.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.