Batch stateful DoFn does not work with runner v2 portable job submission
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When using Stateful DoFn in batch pipeline with dataflow runner v2, workflow will fail with
```
Workflow failed. Causes: S08:Reshuffle/ReshufflePerKey/GroupByKey/Read+Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Reshuffle/RemoveRandomKeys+Do/Reshard/Write
failed., The job failed because a work item has failed 4 times. Look in previous log entries for the
cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors.
The work item was attempted on these workers:
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 20).,
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 20).,
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 20).,
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 20).
```
Imported from Jira [BEAM-11793](https://issues.apache.org/jira/browse/BEAM-11793). Original Jira may contain additional context.
Reported by: yichi.
Contributor guide
Assessment
This issue has not been assessed yet.