apache / apache/beam

WriteTables leaves behind temporary tables on job failure

Open
#21,168 0 comments 0 reactions 0 assignees View on GitHub
extensions gcp improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I'm running a job that writes to a BigQuery table using `BigQueryIO.writeTableRows().to(
new SerializableFunction, TableDestination\>)`.
 
I'm noticing that when my job fails, it leaves behind temporary tables (`beam_bq_job_LOAD_*`) in the destination dataset. These tables are created by load jobs started here:
 
[https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L273-L284)](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L273-L284),)
 
I'd like to specify a temporary dataset for these load job result tables, but I don't see a way to specify one using the Java SDK. It seems like the load job destination is inferred by changing the table id of the final destination:
 
[https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L255](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L255)
 
which makes me think that the configuration I want to set doesn't exist. Is there a workaround to avoid having these tables be left around when the job fails? Could the option be added?

Imported from Jira [BEAM-12986](https://issues.apache.org/jira/browse/BEAM-12986). Original Jira may contain additional context.
Reported by: Wyszynski.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.