apache / apache/beam

DoFn invocation has wrong argument assignment

Open
#19,631 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I've added this test to DoFnProcessTest:
```

class DoFnProcessAllDefaults(DoFn):
def process(self, element=1, timestamp=DoFn.TimestampParam,
side_input=3):
print('element={} timestamp={} side_input={}'.format(
element,
timestamp, side_input))

pipeline_options = PipelineOptions()
with TestPipeline(options=pipeline_options)
as p:
_ = (p
| beam.Create([5])
| beam.ParDo(DoFnProcessAllDefaults(),
2))

```

The print output is:
```

element=5 timestamp=2 side_input=Timestamp(-9223372036854.775000)

```

If the default value for `element` is removed:
```

element=5 timestamp=Timestamp(-9223372036854.775000) side_input=2

```

Bug is in PerWindowInvoker.__init__, found while working on type hints related code.
The issue is with args_to_pick, which goes to -1 in the first case.
Note that prototypes like process(element=1) and process(element=1, timestamp=DoFn.TimestampParam) work fine, so there could be existing pipelines that have such process() methods.

[~robertwb][~altay]

Imported from Jira [BEAM-7867](https://issues.apache.org/jira/browse/BEAM-7867). Original Jira may contain additional context.
Reported by: udim.

Contributor guide

Open the contributing guide

Research direction

Start with the added DoFnProcessTest case and inspect PerWindowInvoker.__init__, focusing on how args_to_pick is calculated. Reproduce the default-argument examples and compare them with the working process(element=1) and process(element=1, timestamp=DoFn.TimestampParam) cases. Done means element, timestamp, and side_input receive the correct values in all shown prototypes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.