apache / apache/beam

dataflow_metrics.py should be able to fully query the metrics

Open
#19,452 0 comments 0 reactions 0 assignees View on GitHub
core new feature P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

dataflow_metrics.py expects a job object to be passed in, it cannot fully query the metrics on its own.

Updating this to use the dataflow_client to also query the job itself (non trivial since the job passed in is a wrapper for the Job proto the dataflow_client obtains, this wrapper is created by dataflow_runner.) Though we may not need any of the extra wrapper features, since we only need access to the graph. But it is unclear if this will work, as the graph DatalowRunner generates is not the same graph as the one queried out.

 

The job graph is use to translate step names

Imported from Jira [BEAM-6833](https://issues.apache.org/jira/browse/BEAM-6833). Original Jira may contain additional context.
Reported by: ajamato@google.com.

Contributor guide

Open the contributing guide

Research direction

Start with dataflow_metrics.py and trace how it receives the job object, then inspect dataflow_client and dataflow_runner to understand the queried Job proto, wrapper, and generated graph. Done means dataflow_metrics.py can query the job itself and use the job graph to translate step names, with the graph mismatch resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.