WriteTables leaves behind temporary tables on job failure
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I'm running a job that writes to a BigQuery table using `BigQueryIO.writeTableRows().to(
new SerializableFunction, TableDestination\>)`.
I'm noticing that when my job fails, it leaves behind temporary tables (`beam_bq_job_LOAD_*`) in the destination dataset. These tables are created by load jobs started here:
[https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L273-L284)](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L273-L284),)
I'd like to specify a temporary dataset for these load job result tables, but I don't see a way to specify one using the Java SDK. It seems like the load job destination is inferred by changing the table id of the final destination:
[https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L255](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L255)
which makes me think that the configuration I want to set doesn't exist. Is there a workaround to avoid having these tables be left around when the job fails? Could the option be added?
Imported from Jira [BEAM-12986](https://issues.apache.org/jira/browse/BEAM-12986). Original Jira may contain additional context.
Reported by: Wyszynski.
Contributor guide
Assessment
This issue has not been assessed yet.