apache / apache/parquet-format

parquet-format and parquet-format-structures defines Util with inconsitent methods provided

Open
#420 2 comments 0 reactions 0 assignees View on GitHub
Priority: Major Type: bug
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

I have been running into a bug due to `parquet-format` and `parquet-format-structures` both defining the `org.apache.parquet.format.Util` class but doing so inconsistently.

Examples of this are several methods which include a `BlockCipher` parameter that are defined from `parquet-format-structures` but not `{}parquet-format{`}. While invoking code that happens to use these, such as `{}org.apache.parquet.hadoop.ParquetFileReader.readFooter{`}, the code will fail if the `parquet-format` happens to be loaded first on the classpath.

Here is an example stack trace for a Scala Spark application.
```java

Caused by: java.lang.NoSuchMethodError: 'org.apache.parquet.format.FileMetaData org.apache.parquet.format.Util.readFileMetaData(java.io.InputStream, org.apache.parquet.format.BlockCipher$Decryptor, byte[])'
at org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1441) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1438) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.format.converter.ParquetMetadataConverter$NoFilter.accept(ParquetMetadataConverter.java:1173) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1438) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:591) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:536) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:530) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:478) ~[parquet_hadoop.jar:1.13.1]
... (my application code invoking the above)
```
Because of issues external to Parquet that I have yet to figure out (a complex Spark and dependency setup), my classpaths are not deterministically ordered and I am unable to pin the `parquet-format-structures` ahead hence why I'm chiming in about this.

Even if that weren't the case, this is a fairly prickly edge to run into as both modules define overlapping classes. `Util` is not the only class that appears to be defined by both, just what I have been focusing on due to this bug.
It appears these methods were introduced in at least 1.12:

**Reporter**: [Joey Pereira](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=legojoey17)

**Note**: *This issue was originally created as [PARQUET-2317](https://issues.apache.org/jira/browse/PARQUET-2317). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

Open the contributing guide

Research direction

Compare the org.apache.parquet.format.Util definitions in parquet-format and parquet-format-structures, focusing on the BlockCipher methods used by ParquetFileReader.readFooter. Reproduce the failure with parquet-format first on the classpath; done means the overlapping definitions are consistent and the NoSuchMethodError no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.