[Bug]: Dataflow worker retries pipeline if FinishBundle blocks for too long
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
While performing load tests for the FhirIO [Import](https://github.com/lnogueir/beam/blob/a7abdc2771098576dacae7c71a8ebf32b77c2ed2/sdks/go/pkg/beam/io/fhirio/import.go#L220) transform I implemented in #22460 for the Go SDK, I noticed that after the resources were successfully imported, the pipeline would restart from the beginning. It would do this 4 times until the pipeline fails with the error: `The job failed because a work item has failed 4 times. Root cause: The worker lost contact with the service.`
My theory is that because the import request performed inside `FinishBundle` takes several minutes to complete (since it is importing millions of resources), the runner thinks it failed and then it retries.
The reason for filing this issue is because a blocking call that takes the same amount of time does not cause this retry loop failure and completes successfully in the Java SDK, so there's a discrepancy there. Also because using a long-running `FinishBundle` method is a Beam design pattern that is widespread and even considered a best practice as described [here](https://cloud.google.com/architecture/e-commerce/patterns/batching-external-calls).
The issue can be reproduced with a dummy pipeline that just sleeps for 20 minutes inside `FinishBundle` or any other `DoFn` lifecycle method. I made these dummy pipelines available so others can easily reproduce the issue themselves: https://github.com/apache/beam/compare/master...lnogueir:beam:dataflowWorkerLiveness
Instructions on how to execute them are commented on the files in the link above.
I have validated that the Java SDK doesn't experience this issue on neither runner V1 nor V2.
CC: @lostluck @jrmccluskey @msbukal @youngoli
### Issue Priority
Priority: 2
### Issue Component
Component: io-go-gcp
Contributor guide
Research direction
The reproduction is in the dataflowWorkerLiveness changes linked from the issue; start with those dummy pipelines and their commented execution instructions, then inspect the Go SDK path around sdks/go/pkg/beam/io/fhirio/import.go:220. Done means a long-running FinishBundle or lifecycle call no longer causes Dataflow to retry the pipeline, while the reproduction completes as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100