ParquetIOIT fails on Spark and Flink
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When run on Spark or Flink remote cluster, ParquetIOIT fails with the following stacktrace:
```
org.apache.beam.sdk.io.parquet.ParquetIOIT > writeThenReadAll FAILED
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.NoSuchMethodError: org.apache.parquet.hadoop.ParquetWriter$Builder.(Lorg/apache/parquet/io/OutputFile;)V
at
org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:66)
at
org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:99)
at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:87)
at
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:116)
at org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:61)
at
org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:350)
at
org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:331)
at org.apache.beam.sdk.io.parquet.ParquetIOIT.writeThenReadAll(ParquetIOIT.java:133)
Caused
by:
java.lang.NoSuchMethodError: org.apache.parquet.hadoop.ParquetWriter$Builder.(Lorg/apache/parquet/io/OutputFile;)V
```
Imported from Jira [BEAM-5164](https://issues.apache.org/jira/browse/BEAM-5164). Original Jira may contain additional context.
Reported by: ŁukaszG.
Contributor guide
Assessment
This issue has not been assessed yet.