_MaterializedDoOutputsTuple doesn't support __getitem__ by integer values
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Consider the following pipeline:
`with beam.Pipeline(..) as p: `
` res = p | ... | beam.Partition(..)`
When res is an `_apache_beam.pvalue.DoOutputsTuple_`, it supports access by `res[0]` and `res["0"]`. However, if res is a `_apache_beam.transforms.ptransform._MaterializedDoOutputsTuple_', integer access isn't supported and we must access as strings, although not very intuitive considering that `_partition_fn_` returns integers.
I'm not familiar with beam internals but briefly looked into the code and I saw that __MaterializedDoOutputsTuple overrides the __getitem__() of DoOutputsTuple and doesn't have the explicit casting ([https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pvalue.py#L225).](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pvalue.py#L225).)
Also looks like [~gildea] already had a related comment there.
Is this on purpose? Can I expect an access-by-int API for Partition() results regardless of whether it was materialized or not?
Imported from Jira [BEAM-5805](https://issues.apache.org/jira/browse/BEAM-5805). Original Jira may contain additional context.
Reported by: zachmoshe.
Contributor guide
Research direction
Start in sdks/python/apache_beam/pvalue.py at DoOutputsTuple.__getitem__ and _MaterializedDoOutputsTuple, then reproduce the difference with a Partition() result. Done means integer and string indexing behave consistently for materialized and non-materialized results, with a regression test covering integer access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100