TypeCheckError due to CoGroupByKey output mis-deduction
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
After upgrading our Python project from 2.31.0 to 2.33.0, we started getting TypeCheckErrors such as
> apache_beam.typehints.decorators.TypeCheckError: Type hint violation for 'all_data/combine_new_and_all': requires `Tuple[Tuple[Any, Any], Dict[str, Iterable[_CombinedEntry]]]` but got `Tuple[Tuple[int, int], Dict[str, List[Union[]]]]` for element
>
> ere the output value of a `CoGroupByKey()` is apparently incorrectly deduced to be a `Dict[str, List[Union[]]]`.
I managed to build a small repro case:
```
import apache_beam as beam
from typing import Dict, Iterable, Tuple
{
"foo": [(42, "foo")],
"bar": [(42, "bar")],
} | beam.CoGroupByKey().with_output_types(Tuple[int, Dict[str, Iterable[str]]])
```
which raises
> apache_beam.typehints.decorators.TypeCheckError: Output type hint violation at CoGroupByKey: expected `Tuple[int, Dict[str, Iterable[str]]]`, got `Tuple[int, Dict[str, List[Union[]]]]`
>
> alternatively, using a TestPipeline:
> ```
import apache_beam as beam
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util
import assert_that, equal_to
from typing import Dict, Iterable, Tuple
with TestPipeline() as p:
actual = {
"foo": p | "create_foo" >> beam.Create([(42, "foo")]),
"bar": p | "create_bar"
>> beam.Create([(42, "bar")]),
} | beam.CoGroupByKey().with_output_types(Tuple[int, Dict[str, Iterable[str]]])
assert_that(actual, equal_to([(42, {"foo": ["foo"], "bar": ["bar"]})]))
```
Oh, and one more thing, about that `Tuple[Any, Any]` from the original error message I posted. We can reproduce that like this:
```
import apache_beam as beam
from typing import Dict, Iterable, NewType, Tuple
key = NewType("key",
int)
{
"foo": [(key(1337), "foo")],
"bar": [(key(1337), "bar")],
} | beam.CoGroupByKey().with_output_types(Tuple[key,
Dict[str, Iterable[str]]])
```
> apache_beam.typehints.decorators.TypeCheckError: Output type hint violation at CoGroupByKey: expected `Tuple[Any, Dict[str, Iterable[str]]]`, got `Tuple[int, Dict[str, List[Union[]]]]`
>
> looks like `NewType` is treated as `Any`? That surprised me.
I could also reproduce the issue in 2.32.0.
Imported from Jira [BEAM-13217](https://issues.apache.org/jira/browse/BEAM-13217). Original Jira may contain additional context.
Reported by: mrwonko.
Contributor guide
Research direction
Run the minimal CoGroupByKey().with_output_types repro first, then trace the Python type-hint deduction for CoGroupByKey, including the NewType case. Add a regression test based on the repro and confirm that both examples accept the declared output types and produce the expected grouped values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100