[Bug]: [Java BQ] Race condition between copy jobs at different triggers in a streaming pipeline
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
There seems to be a race condition between copy jobs at different triggers in a streaming pipeline. Sometimes, the second trigger's copy job starts running before the first trigger's copy job. The second trigger's copy job runs and succeeds in writing data to the table; then the first trigger's copy job runs.
#### Why is this a problem?
We attach whatever dispositions the user sets to the first copy job, because we assume it will run first. All copy jobs after that are fixed to `WRITE_APPEND` and `CREATE_NEVER` because we are continuously writing data to an existing table.
The problem comes in when the user sets a `WRITE_EMPTY` disposition. This gets attached to the first copy job. When the race condition takes effect and copy job#2 runs first, we end up with data in the table. Then when copy job#1 runs, it fails because of its `WRITE_EMPTY` disposition.
Example below: In the following logs, `beam_bq_job_COPY_...510c8e70fecb762c7_00000-0` is copy jobs ID of the first trigger, `beam_bq_job_COPY_...510c8e70fecb762c7_00001-0` is copy jobs ID of the second trigger. The suffix of these names is tableHash_**paneNumber**-retryIndex. We see copy jobs at pane 1 starting earlier than pane 0. Not sure what is causing this to happen.
Example logs
INFO: Job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0 pending. retrying.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
INFO: Load job beam_bq_job_TEMP_TABLE_LOAD_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_05014b97910b4635a89409bd024d057d_1af13eb133f5d1e510c8e70fecb762c7_00001_00000-0 succeeded. Statistics: null
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Job beam_bq_job_TEMP_TABLE_LOAD_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_05014b97910b4635a89409bd024d057d_1af13eb133f5d1e510c8e70fecb762c7_00001_00000-0 completed successfully.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Waiting for jobs to complete.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Waiting for jobs to complete.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0 pending. retrying.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.WriteRename lambda$startCopy$57215ec2$1
INFO: Starting copy job for table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=dataset-id, projectId=project-id, tableId=table-id}} using GenericData{classInfo=[jobId, location, projectId], {jobId=beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0, projectId=project-id}}, job id iteration 0
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
INFO: job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0 started
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.WriteRename lambda$startCopy$57215ec2$1
INFO: Starting copy job for table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=dataset-id, projectId=project-id, tableId=table-id}} using GenericData{classInfo=[jobId, location, projectId], {jobId=beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0, projectId=project-id}}, job id iteration 0
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
INFO: job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0 started
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
INFO: Load job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0 succeeded. Statistics: null
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager waitForDone
INFO: Job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0 completed successfully.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers getRetryJobId
INFO: job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0 is failed. Checking the next job id
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers getRetryJobId
INFO: job id beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-1 not found, so retrying with that id
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
WARNING: Load job beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0 failed, will retry: {
"errorResult" : { },
"state" : "FAILED"
}. Next job id beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-1
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.