apache / apache/beam

Push metrics to a backend in an runner agnostic way

Open
#18,594 1 comment 0 reactions 0 assignees View on GitHub
core java new feature P3 runner-extensions-metrics
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

The idea is to avoid relying on the runners to provide access to the metrics (either at the end of the pipeline or while it runs) because they don't have all the same capabilities towards metrics (e.g. spark runner configures sinks like csv, graphite or in memory sinks using the spark engine conf). The target is to push the metrics in the common runner code so that no matter the chosen runner, a user can get his metrics out of beam.
Here is the link to the discussion thread on the dev ML: https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
And the design doc:
https://s.apache.org/runner_independent_metrics_extraction

Imported from Jira [BEAM-3310](https://issues.apache.org/jira/browse/BEAM-3310). Original Jira may contain additional context.
Reported by: echauchot.

Contributor guide

Open the contributing guide

Research direction

Start with the linked design document and dev mailing-list discussion to understand the proposed runner-independent metrics flow. Then locate the common runner code described in the issue and determine how metrics should be pushed regardless of runner capabilities. Done means users can retrieve metrics consistently without relying on runner-specific sinks.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.