apache / apache/beam

Python SDK creates job graphs with duplicated states when using fn_api execution mode.

Open
#19,264 0 comments 0 reactions 0 assignees View on GitHub
bug harness P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

We observed this on apache_beam.examples.wordcount with Dataflow runner.

The graph for this wordcount job contains two steps with the same name "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1".

```

...
{
"kind": "PAR_DO_KIND",
"id": "s41",
"name": "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1",

"displayData": [
{
"key": "fn",
"namespace": "apache_beam.transforms.core.ParDo",

"strValue": "apache_beam.transforms.core.CallableWrapperDoFn",
"shortStrValue":
"CallableWrapperDoFn",
"label": "Transform Function"
},
{

"key": "fn",
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn",

"strValue": "\u003clambda\u003e",
"label": "Transform Function"
}

],
"outputCollectionName": [
"write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1.out0"

],
"inputCollectionName": [
"write/Write/WriteImpl/Extract.out0"

]
},
...
{
"kind": "PAR_DO_KIND",
"id": "s31",
"name": "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1",

"displayData": [
{
"key": "fn",
"namespace": "apache_beam.transforms.core.ParDo",

"strValue": "apache_beam.transforms.core.CallableWrapperDoFn",
"shortStrValue":
"CallableWrapperDoFn",
"label": "Transform Function"
},
{

"key": "fn",
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn",

"strValue": "\u003clambda\u003e",
"label": "Transform Function"
}

],
"outputCollectionName": [
"write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1.out0"

],
"inputCollectionName": [
"write/Write/WriteImpl/Extract.out0"

]
},
...

```

CC: [~foegler] [~altay] [~robertwb]

Imported from Jira [BEAM-6584](https://issues.apache.org/jira/browse/BEAM-6584). Original Jira may contain additional context.
Reported by: tvalentyn.

Contributor guide

Open the contributing guide

Research direction

Start with apache_beam.examples.wordcount using the Dataflow runner and fn_api execution mode, then inspect the generated job graph for duplicated state names. Done means the graph no longer contains duplicate steps such as write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.