apache / apache/beam

BigQueryIO failed to load data to temp table when withSchemaUpdateOptions is set.

Open
#21,105 1 comment 0 reactions 0 assignees View on GitHub
bug gcp io java P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

`withSchemaUpdateOptions` is only supported with WRITE_APPEND or WRITE_TRUNCATE with paritioned table. This is noted in the documentation:

[https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1965](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1965)

When the temp files are spilt to multiple load jobs, the write disposition is set to WRITE_TRUNCATE for the temporary tables in favor of retries.

[https://github.com/apache/beam/blob/a8a8b3ab1292ff408940ba4430c95bc754341589/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L219](https://github.com/apache/beam/blob/a8a8b3ab1292ff408940ba4430c95bc754341589/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L219)

And the loading would fail with error 

"Schema update options should only be specified with WRITE_APPEND disposition, or with WRITE_TRUNCATE disposition on a table partition."

 

Imported from Jira [BEAM-12482](https://issues.apache.org/jira/browse/BEAM-12482). Original Jira may contain additional context.
Reported by: sychen.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.