BigQueryIO.write.withExtendedErrorInfo() does not catch missing tables (and maybe datasets)
Open
bug
gcp
io
java
P3
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
In beam versions from 2.7.0 to 2.9.0 a missing table produces an exception in Dataflow that .getFailedInsertsWithErr() doesn't catch.
We get a GoogleJsonResponseException 404.
In beam versions 2.10.0 or 2.11.0 the step just gets silently stuck (logging every 5mins).
I would expect this error to be "catchable" just like a missing field in a table is.
I haven't checked the behaviour with a missing dataset yet, but I would like to be able to take action on this kind of error as well.
Imported from Jira [BEAM-7022](https://issues.apache.org/jira/browse/BEAM-7022). Original Jira may contain additional context.
Reported by: Pierig.
Contributor guide
Assessment
This issue has not been assessed yet.