Python SDK creates job graphs with duplicated states when using fn_api execution mode.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
We observed this on apache_beam.examples.wordcount with Dataflow runner.
The graph for this wordcount job contains two steps with the same name "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1".
```
...
{
"kind": "PAR_DO_KIND",
"id": "s41",
"name": "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1",
"displayData": [
{
"key": "fn",
"namespace": "apache_beam.transforms.core.ParDo",
"strValue": "apache_beam.transforms.core.CallableWrapperDoFn",
"shortStrValue":
"CallableWrapperDoFn",
"label": "Transform Function"
},
{
"key": "fn",
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn",
"strValue": "\u003clambda\u003e",
"label": "Transform Function"
}
],
"outputCollectionName": [
"write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1.out0"
],
"inputCollectionName": [
"write/Write/WriteImpl/Extract.out0"
]
},
...
{
"kind": "PAR_DO_KIND",
"id": "s31",
"name": "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1",
"displayData": [
{
"key": "fn",
"namespace": "apache_beam.transforms.core.ParDo",
"strValue": "apache_beam.transforms.core.CallableWrapperDoFn",
"shortStrValue":
"CallableWrapperDoFn",
"label": "Transform Function"
},
{
"key": "fn",
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn",
"strValue": "\u003clambda\u003e",
"label": "Transform Function"
}
],
"outputCollectionName": [
"write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1.out0"
],
"inputCollectionName": [
"write/Write/WriteImpl/Extract.out0"
]
},
...
```
CC: [~foegler] [~altay] [~robertwb]
Imported from Jira [BEAM-6584](https://issues.apache.org/jira/browse/BEAM-6584). Original Jira may contain additional context.
Reported by: tvalentyn.
Contributor guide
Research direction
Start with apache_beam.examples.wordcount using the Dataflow runner and fn_api execution mode, then inspect the generated job graph for duplicated state names. Done means the graph no longer contains duplicate steps such as write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100