apache / apache/beam

PackageUtil should retry check for whether file is already staged

Open
#18,663 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

The code to stage files retries the upload, but the existence (and size) check is attempted only once:

https://github.com/apache/beam/blob/dc3a664e3b4b4f63500954aae25dce0cb5f8b314/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java#L193

This causes post-commit tests to fail sometimes:

https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4632/testReport/junit/org.apache.beam.sdk.io/BoundedReadFromUnboundedSourceTest/testDedup/

https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4636/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyMultimapSideInput/

We should retry this check as well.

Imported from Jira [BEAM-3408](https://issues.apache.org/jira/browse/BEAM-3408). Original Jira may contain additional context.
Reported by: herohde.

Contributor guide

Open the contributing guide

Research direction

Inspect runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java around line 193, then compare the existing upload retry behavior with the one-time existence and size check. Run the named Dataflow post-commit tests, including BoundedReadFromUnboundedSourceTest/testDedup and ViewTest/testEmptyMultimapSideInput; done means the check retries and those intermittent failures no longer occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.