apache / apache/beam

[Bug]: Dataflow worker retries pipeline if FinishBundle blocks for too long

Open
#22,610 2 comments 0 reactions 0 assignees View on GitHub
bug gcp go io P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

While performing load tests for the FhirIO [Import](https://github.com/lnogueir/beam/blob/a7abdc2771098576dacae7c71a8ebf32b77c2ed2/sdks/go/pkg/beam/io/fhirio/import.go#L220) transform I implemented in #22460 for the Go SDK, I noticed that after the resources were successfully imported, the pipeline would restart from the beginning. It would do this 4 times until the pipeline fails with the error: `The job failed because a work item has failed 4 times. Root cause: The worker lost contact with the service.`

My theory is that because the import request performed inside `FinishBundle` takes several minutes to complete (since it is importing millions of resources), the runner thinks it failed and then it retries.

The reason for filing this issue is because a blocking call that takes the same amount of time does not cause this retry loop failure and completes successfully in the Java SDK, so there's a discrepancy there. Also because using a long-running `FinishBundle` method is a Beam design pattern that is widespread and even considered a best practice as described [here](https://cloud.google.com/architecture/e-commerce/patterns/batching-external-calls).

The issue can be reproduced with a dummy pipeline that just sleeps for 20 minutes inside `FinishBundle` or any other `DoFn` lifecycle method. I made these dummy pipelines available so others can easily reproduce the issue themselves: https://github.com/apache/beam/compare/master...lnogueir:beam:dataflowWorkerLiveness

Instructions on how to execute them are commented on the files in the link above.

I have validated that the Java SDK doesn't experience this issue on neither runner V1 nor V2.

CC: @lostluck @jrmccluskey @msbukal @youngoli

### Issue Priority

Priority: 2

### Issue Component

Component: io-go-gcp

Contributor guide

Open the contributing guide

Research direction

The reproduction is in the dataflowWorkerLiveness changes linked from the issue; start with those dummy pipelines and their commented execution instructions, then inspect the Go SDK path around sdks/go/pkg/beam/io/fhirio/import.go:220. Done means a long-running FinishBundle or lifecycle call no longer causes Dataflow to retry the pipeline, while the reproduction completes as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.