FlinkDistributionGauge and FlinkGauge metrics are exported as zero to Prometheus when using any Flink's PrometheusReporter
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
To be honest I'm really lost on this one, let me explain the issue:
Beam has its own metrics types (org/apache/beam/sdk/metrics/Metrics.java) \-counter, distribution, and gauge\-, and, depending on the runner, wraps them into their corresponding runner types. For example, for Flink, Beam is wrapping its Gauge type into a class called FlinkGauge which extends a Gauge.
Also, Beam's Distribution metric its wrapped into a Flink's Gauge, where DistributionResult is a Beam type containing min,max,sum,count.
Then, if you are using Flink, and you want to export those metrics to Prometheus, using flink-metrics-prometheus, you will see that they are always zero, and, if you set DEBUG log level for "org.apache.flink.metrics.prometheus" package, you will see errors like following ones:
```
2020-09-18 06:27:04,387 DEBUG Invalid type for Gauge org.apache.beam.runners.flink.metrics.FlinkMetricContainer$FlinkDistributionGauge@30211d3f:
org.apache.beam.sdk.metrics.AutoValue_DistributionResult, only number types and booleans are supported
by this reporter.
2020-09-18 06:27:04,394 DEBUG Invalid type for Gauge org.apache.beam.runners.flink.metrics.FlinkMetricContainer$FlinkGauge@2ad1562:
org.apache.beam.sdk.metrics.AutoValue_GaugeResult, only number types and booleans are supported by this
reporter.
```
Which is really weird, because if you check the source code of AbstractPrometheusReporter, you can see that is taking the metric value from Flink's Gauge using getValue():
https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/AbstractPrometheusReporter.java#L225
And FlinkGauge.getValue() should return a long instead of org.apache.beam.sdk.metrics.AutoValue_GaugeResult. So I don't understand what is happening there to be honest. May be AutoValue mechanism is messing things up?
Imported from Jira [BEAM-10928](https://issues.apache.org/jira/browse/BEAM-10928). Original Jira may contain additional context.
Reported by: ivansjg.
Contributor guide
Research direction
Start with org/apache/beam/sdk/metrics/Metrics.java and the FlinkMetricContainer FlinkGauge and FlinkDistributionGauge implementations. Compare their getValue() behavior with AbstractPrometheusReporter at the linked Flink entry point, then reproduce the reporter output and verify that gauge and distribution metrics are exported with their actual values rather than zero.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, prometheus
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100