apache / apache/beam

_MaterializedDoOutputsTuple doesn't support __getitem__ by integer values

Open
#19,154 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Consider the following pipeline:

`with beam.Pipeline(..) as p:  `

`  res = p | ... | beam.Partition(..)`

 

When res is an `_apache_beam.pvalue.DoOutputsTuple_`, it supports access by `res[0]` and `res["0"]`. However, if res is a `_apache_beam.transforms.ptransform._MaterializedDoOutputsTuple_', integer access isn't supported and we must access as strings, although not very intuitive considering that `_partition_fn_` returns integers.

 

I'm not familiar with beam internals but briefly looked into the code and I saw that __MaterializedDoOutputsTuple overrides the __getitem__() of DoOutputsTuple and doesn't have the explicit casting ([https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pvalue.py#L225).](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pvalue.py#L225).)

Also looks like [~gildea] already had a related comment there.

 

Is this on purpose? Can I expect an access-by-int API for Partition() results regardless of whether it was materialized or not?

Imported from Jira [BEAM-5805](https://issues.apache.org/jira/browse/BEAM-5805). Original Jira may contain additional context.
Reported by: zachmoshe.

Contributor guide

Open the contributing guide

Research direction

Start in sdks/python/apache_beam/pvalue.py at DoOutputsTuple.__getitem__ and _MaterializedDoOutputsTuple, then reproduce the difference with a Partition() result. Done means integer and string indexing behave consistently for materialized and non-materialized results, with a regression test covering integer access.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.