[Feature Request]: In Python how to assign types to multi-output tags?
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
I have a multi-output DoFn:
```py
class DoFn1:
def process(self, row) -> Iterable[Union[Dict[str, Any], pvalue.TaggedOutput]]:
if something:
yield some_dict(...)
else:
yield pvalue.TaggedOutput("bad", ...)
```
And another DoFn that consumes its outputs
```py
class DoFn2:
def process(self, row: Dict[str, Any]) -> Iterable[...]:
if something:
yield some_dict(...)
else:
yield pvalue.TaggedOutput("bad", ...)
```
And then when I use it like this:
```py
pcoll = ...
pcoll = pcoll | "dofn1" >> beam.ParDo(DoFn1()).with_outputs(
"bad",
main="good",
)
pcoll["good"] | "dofn2" >> beam.ParDo(DoFn2())
```
I get an error that looks like this:
```
apache_beam.typehints.decorators.TypeCheckError: Type hint violation for 'dofn2': requires Dict[str, Any]
but got Union[Dict[str, Any], TaggedOutput] for row
```
How can I tell Beam type checking that the `good` tag gives `Dict[str, Any]`, and `bad` gives `TaggedOutput`? Am I forced to propagate the tagged output type hints?
Also asked in SO here if not appropriate for Github: https://stackoverflow.com/questions/76151787/in-apache-beam-dataflow-multi-output-dofns-how-do-you-assign-type-hints-to-spec
I found a related discussion here: https://github.com/apache/beam/pull/9810
### Issue Priority
Priority: 3 (nice-to-have improvement)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.