PackageUtil should retry check for whether file is already staged
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
The code to stage files retries the upload, but the existence (and size) check is attempted only once:
https://github.com/apache/beam/blob/dc3a664e3b4b4f63500954aae25dce0cb5f8b314/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java#L193
This causes post-commit tests to fail sometimes:
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4632/testReport/junit/org.apache.beam.sdk.io/BoundedReadFromUnboundedSourceTest/testDedup/
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4636/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyMultimapSideInput/
We should retry this check as well.
Imported from Jira [BEAM-3408](https://issues.apache.org/jira/browse/BEAM-3408). Original Jira may contain additional context.
Reported by: herohde.
Contributor guide
Research direction
Inspect runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java around line 193, then compare the existing upload retry behavior with the one-time existence and size check. Run the named Dataflow post-commit tests, including BoundedReadFromUnboundedSourceTest/testDedup and ViewTest/testEmptyMultimapSideInput; done means the check retries and those intermittent failures no longer occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100