apache / apache/parquet-format
parquet-format and parquet-format-structures defines Util with inconsitent methods provided
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
I have been running into a bug due to `parquet-format` and `parquet-format-structures` both defining the `org.apache.parquet.format.Util` class but doing so inconsistently.
Examples of this are several methods which include a `BlockCipher` parameter that are defined from `parquet-format-structures` but not `{}parquet-format{`}. While invoking code that happens to use these, such as `{}org.apache.parquet.hadoop.ParquetFileReader.readFooter{`}, the code will fail if the `parquet-format` happens to be loaded first on the classpath.
Here is an example stack trace for a Scala Spark application.
```java
Caused by: java.lang.NoSuchMethodError: 'org.apache.parquet.format.FileMetaData org.apache.parquet.format.Util.readFileMetaData(java.io.InputStream, org.apache.parquet.format.BlockCipher$Decryptor, byte[])'
at org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1441) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1438) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.format.converter.ParquetMetadataConverter$NoFilter.accept(ParquetMetadataConverter.java:1173) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1438) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:591) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:536) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:530) ~[parquet_hadoop.jar:1.13.1]
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:478) ~[parquet_hadoop.jar:1.13.1]
... (my application code invoking the above)
```
Because of issues external to Parquet that I have yet to figure out (a complex Spark and dependency setup), my classpaths are not deterministically ordered and I am unable to pin the `parquet-format-structures` ahead hence why I'm chiming in about this.
Even if that weren't the case, this is a fairly prickly edge to run into as both modules define overlapping classes. `Util` is not the only class that appears to be defined by both, just what I have been focusing on due to this bug.
It appears these methods were introduced in at least 1.12:
**Reporter**: [Joey Pereira](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=legojoey17)
**Note**: *This issue was originally created as [PARQUET-2317](https://issues.apache.org/jira/browse/PARQUET-2317). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
Research direction
Compare the org.apache.parquet.format.Util definitions in parquet-format and parquet-format-structures, focusing on the BlockCipher methods used by ParquetFileReader.readFooter. Reproduce the failure with parquet-format first on the classpath; done means the overlapping definitions are consistent and the NoSuchMethodError no longer occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100