apache / apache/beam

Add tests for exporting Beam Metrics to Flink Metrics

Open
#18,323 0 comments 0 reactions 0 assignees View on GitHub
flink improvement P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

There are currently no tests that verify that metrics that are reported using the Beam Metrics API are forwarded to Flink and a `MetricReporter`.

A test for this would have to manually configure a Flink "Mini Cluster", as in
```

// start also a re-usable Flink mini cluster
flink = new LocalFlinkMiniCluster(getFlinkConfiguration(),
false);
flink.start();
flinkPort = flink.getLeaderRPCPort();

```

with `getFlinkConfiguration()`:
```

protected static Configuration getFlinkConfiguration() {
Configuration flinkConfig = new Configuration();

flinkConfig.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 1);
flinkConfig.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS,
8);
flinkConfig.setInteger(ConfigConstants.TASK_MANAGER_MEMORY_SIZE_KEY, 16);
flinkConfig.setString(ConfigConstants.RESTART_STRATEGY_FIXED_DELAY_DELAY,
"0 s");
flinkConfig.setString(ConfigConstants.METRICS_REPORTERS_LIST, "my_reporter");
flinkConfig.setString(ConfigConstants.METRICS_REPORTER_PREFIX
+ "my_reporter." + ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, MyTestReporter.class.getName());

return flinkConfig;
}

```

where `MyTestReporter` is a `MetricReporter` that stores metrics being reported to it so we can verify that they are there after the job finishes.

Running a Pipeline on the mini cluster should be possible by specifying "localhost" and the port we received as a cluster endpoint.

Imported from Jira [BEAM-2056](https://issues.apache.org/jira/browse/BEAM-2056). Original Jira may contain additional context.
Reported by: aljoscha.

Contributor guide

Open the contributing guide

Research direction

The issue names no target files or existing test entry point. Start with the shown LocalFlinkMiniCluster setup, getFlinkConfiguration(), and the localhost cluster endpoint; define a MetricReporter that retains reported metrics and run a Pipeline. Done means the test verifies Beam Metrics are forwarded to the Flink MetricReporter after the job finishes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.