Add integration testing for BQ Storage API write modes
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
We added STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE BQ sink write modes recently [1] but seems like this is not covered by integration testing today.
This could result in significant breakages that creep into Beam releases (for example, https://github.com/apache/beam/pull/16599).
These write modes should be regularly integration tested using Jenkins for batch/streaming.
[1] https://github.com/apache/beam/blob/41d585f82b10195f758d14e3a54076ea1f05aa75/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1761
Imported from Jira [BEAM-13747](https://issues.apache.org/jira/browse/BEAM-13747). Original Jira may contain additional context.
Reported by: chamikara.
Contributor guide
Research direction
Start with the linked write-mode definitions in sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java, then locate the existing BigQuery sink integration-test and Jenkins entry points. Add regular coverage for STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE in both batch and streaming workflows, with Jenkins runs demonstrating the modes remain functional.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- cloud, data-engineering, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100