[Feature Request]: Use Spark 3 plugin framework to support metrics for Spark runner
Open
new feature
P2
runners
spark
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
Currently the metrics integration requires users to configure a custom, Beam-specific Spark metrics sink which is hard to understand.
At least for Spark 3, this should be based on the Spark plugin framework to make the integration more seamless.
This also requires BeamMetricSet to become an actual MetricRegistry (instead of a Gauge).
### Issue Priority
Priority: 2
### Issue Component
Component: runner-spark
Contributor guide
Assessment
This issue has not been assessed yet.