Add tests for exporting Beam Metrics to Flink Metrics
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
There are currently no tests that verify that metrics that are reported using the Beam Metrics API are forwarded to Flink and a `MetricReporter`.
A test for this would have to manually configure a Flink "Mini Cluster", as in
```
// start also a re-usable Flink mini cluster
flink = new LocalFlinkMiniCluster(getFlinkConfiguration(),
false);
flink.start();
flinkPort = flink.getLeaderRPCPort();
```
with `getFlinkConfiguration()`:
```
protected static Configuration getFlinkConfiguration() {
Configuration flinkConfig = new Configuration();
flinkConfig.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 1);
flinkConfig.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS,
8);
flinkConfig.setInteger(ConfigConstants.TASK_MANAGER_MEMORY_SIZE_KEY, 16);
flinkConfig.setString(ConfigConstants.RESTART_STRATEGY_FIXED_DELAY_DELAY,
"0 s");
flinkConfig.setString(ConfigConstants.METRICS_REPORTERS_LIST, "my_reporter");
flinkConfig.setString(ConfigConstants.METRICS_REPORTER_PREFIX
+ "my_reporter." + ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, MyTestReporter.class.getName());
return flinkConfig;
}
```
where `MyTestReporter` is a `MetricReporter` that stores metrics being reported to it so we can verify that they are there after the job finishes.
Running a Pipeline on the mini cluster should be possible by specifying "localhost" and the port we received as a cluster endpoint.
Imported from Jira [BEAM-2056](https://issues.apache.org/jira/browse/BEAM-2056). Original Jira may contain additional context.
Reported by: aljoscha.
Contributor guide
Research direction
The issue names no target files or existing test entry point. Start with the shown LocalFlinkMiniCluster setup, getFlinkConfiguration(), and the localhost cluster endpoint; define a MetricReporter that retains reported metrics and run a Pipeline. Done means the test verifies Beam Metrics are forwarded to the Flink MetricReporter after the job finishes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100