Python Beam job is successful regardless of the result of the BigQuery load job (for the BigQuery File Loads)
Open
bug
core
gcp
io
P3
python
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
We are using the beam python sdk 2.16.0 and have noticed since upgrading to 2.16.0 that beam jobs are exiting successfully even though the BQ load job fails. We can see the load job failing in the Bigquery UI while the beam jobs running on either the DirectRunner or the DataflowRunner were successful.
Note we are using the BigQuery File Loads (which is enabled via use_beam_bq_sink)
I have verified that downgrading to 2.15.0 fixes this issue so it must be a newly introduced bug.
Imported from Jira [BEAM-8485](https://issues.apache.org/jira/browse/BEAM-8485). Original Jira may contain additional context.
Reported by: rgruener.
Contributor guide
Assessment
This issue has not been assessed yet.