apache / apache/beam

Python SDK: Unable to obtain the PCollection for output tags which are not consumed by a downstream step.

Open
#19,436 0 comments 0 reactions 0 assignees View on GitHub
harness new feature P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I noticed that we are not able to convert the output tag****transform to the pcollection name for metrics (element count/mean byte count), if the Pcollections for the outputed tags are not consumed by a downstream step.

This isn't critical as (1) Arguably there is no pcollection at all. (2) Output but not consumed PCollections are not critical to count metrics on as those can be optomized away entirely (No need to do any work, collect metrics, etc. for an unconsumed pcollection).

However, we are able to count this, but we are unable to assign a pcollection name for it, as in this case there is no information about that output tag defined in the bundle descriptor. The alternative fix is to make sure that its always available, even if not consumed.

Pablo and I looked into this a bit, and he believed it would be possible in pvalue.py's 

DoOutputsTuple class. This fix would require calling __getitem__ on all tags to initialize them properly. However, I had some trouble doing this, as this class is a bit strange since it overrides __getattr__. I found weird behaviors when adding functionality to this code. I don't really get how the code functions today, as its own instance variable usage should trigger the custom __getattr__ code, yet we seem to be using these attrs normally with self.X usages.

Imported from Jira [BEAM-7026](https://issues.apache.org/jira/browse/BEAM-7026). Original Jira may contain additional context.
Reported by: ajamato@google.com.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.