Spark warns on SparkBeamMetric already registered
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Spark runner in some environments (Spark 3 and Databricks) reports a warning on Unknown metric type. The pipeline works well and the metrics appear correct but the warning seems to be a problem of the runner.
>
> 2021-03-23T15:10:02,222] [INFO ] o.a.s.m.MetricsSystem - Metrics already registered
java.lang.IllegalArgumentException: Unknown metric type: class org.apache.beam.runners.spark.metrics.SparkBeamMetric
at com.codahale.metrics.MetricRegistry.notifyListenerOfAddedMetric(MetricRegistry.java:532) ~[spark-3.0.1-yarn-shuffle.jar:?]
at com.codahale.metrics.MetricRegistry.addListener(MetricRegistry.java:363) ~[spark-3.0.1-yarn-shuffle.jar:?]
at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:93) ~[spark-3.0.1-yarn-shuffle.jar:?]
at org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:165) [spark-core_2.12-3.0.1.jar:3.0.1]
at org.apache.beam.runners.spark.SparkRunner.registerMetricsSource(SparkRunner.java:275) [beam-runners-spark-2.27.0.jar:?]
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:250) [beam-runners-spark-2.27.0.jar:?]
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:92) [beam-runners-spark-2.27.0.jar:?]
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322) [beam-sdks-java-harness-2.27.0.jar:?]>
Imported from Jira [BEAM-12032](https://issues.apache.org/jira/browse/BEAM-12032). Original Jira may contain additional context.
Reported by: iemejia.
Contributor guide
Assessment
This issue has not been assessed yet.