apache / apache/beam

Dataflow Jobs meet Cloud Label Spec

Open
#18,628 0 comments 0 reactions 0 assignees View on GitHub
dataflow improvement P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Dataflow Job Names should meet Cloud Label Specifications:

[https://cloud.google.com/resource-manager/docs/creating-managing-labels](https://cloud.google.com/resource-manager/docs/creating-managing-labels)

This will allow Dataflow users to organize resources according to the Dataflow Job using those resources.  The Dataflow Service will be able to automatically associate accessed and consumed resources and associated metrics to support streamlined monitoring and analysis of these resources, in particular GCE instance usage metrics.

Potential Issues:  Some current users appear to encode a lot of data in their job names.  Need to identify if applying this restriction breaks any necessary use cases and verify an alternative is available before proceeding.

 

Imported from Jira [BEAM-3808](https://issues.apache.org/jira/browse/BEAM-3808). Original Jira may contain additional context.
Reported by: foegler.

Contributor guide

Open the contributing guide

Research direction

No source file or test is identified in the issue. Start by locating the Dataflow job-name handling entry point and compare its behavior with the linked Cloud Label specification; determine whether existing encoded names would break and document an acceptable alternative before defining the change.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.