apache / apache/beam

[Bug]: [Java BQ] Reduce the number of retries for streaming Storage API writes

Open
#27,385 3 comments 0 reactions 0 assignees View on GitHub
awaiting triage bug io java P2 stale
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Streaming Storage Write API has a RetryManager with up to 1000 retries and a back-off restricted between 1-10 seconds:
https://github.com/apache/beam/blob/7eaef18bcbfb48fb112884564b1362f638fe5834/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWritesShardedRecords.java#L740-L741

Maybe we should revisit these decisions as 1000 retries might not be needed in a streaming pipeline. In the Dataflow case, if we fail earlier, the runner will retry the bundle as many times as needed. Also, are there any errors that warrant us waiting more than 10 seconds before retrying?

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.