apache / apache/beam

Antipattern will break legacy dataflow python pipelines if a new cy_combiner is added and used in the python counter_factory

Open
#19,433 0 comments 0 reactions 0 assignees View on GitHub
dataflow new feature P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

There are two maps keyed by the cy_combiner object type in dataflow/internal/apiclient.py
structured_counter_translations and counter_translations 

The only way to make changes today without breaking the legacy python dataflow runner harness.  

(1) Introduce the new cy_combiner and modify the apiclient.py's maps which are keyed by the new types in a first PR
(2) Import and release the dataflow containers
(3) Add your code to use the new cy_combiner in the beam SDK

Prior to step 2, you cannot even run dataflow pipelines using your PR. As you will hit a KeyError when the cy_combiner type is looked up in the api_client maps.

Naturally a non dataflow contributor will just try to use the cy_combiner and unexpectedly break the legacy python dataflow runner harness.  

One solution is to catch the key error and log a warning that the metric is dropped, rather than failing the pipeline.

Imported from Jira [BEAM-7050](https://issues.apache.org/jira/browse/BEAM-7050). Original Jira may contain additional context.
Reported by: ajamato@google.com.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.