apache / apache/beam

[Bug]: `Tuple` is confused for a `NamedTuple` across Python versions (3.9 vs 3.10).

Open
#24,685 7 comments 1 reaction 0 assignees View on GitHub
bug P2 python types
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

To me, this is a really interesting corner case! Here's what's going on:

I'm updating an operator in XArray-Beam https://github.com/google/xarray-beam/pull/69. I have a type that I've defined as follows:
```
Chunk = Union[Tuple[Key, xarray.Dataset], Tuple[Key, Tuple[xarray.Dataset, ...]]]
```
In versions of Python before 3.10, this causes no problems. However, in 3.10 (and presumably onward), this type leads to a error:
```
ValueError: expecting type typing.Tuple[xarray_beam._src.core.Key, xarray.core.dataset.Dataset] to have arity 0, had arity 2 instead
```

I've looked a bit in depth at the sources, and I think this is the root cause of the issue:
https://github.com/apache/beam/blob/4886bdf08fd057a0f8a4e99ab30453089654188d/sdks/python/apache_beam/typehints/native_type_compatibility.py#L108

This function, which checks if the tuple has annotations to test if it's a named tuple, doesn't behave as I'd expect. In Python 3.10, it treats `Tuple` as a `NamedTuple` (which it is not). Further, when I change the type annotation to `tuple`, no errors occur. It seems like the recent addition of type-annotations with built-ins will require how `match_is_named_tuple` is implemented:

- https://docs.python.org/3/library/typing.html#typing.Tuple
- https://peps.python.org/pep-0585/

As a workaround, users can use the `tuple` tuple annotation via `__futures__` for Python 3.7 and later.
https://peps.python.org/pep-0585/#implementation

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.