Update BigQueryIO to support schema auto detection when executing load jobs
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
BigQuery supports detecting unspecified schemas automatically when executing load jobs: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load
By default this is false and Dataflow does not set this to true when creating load jobs: https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L228
Probably we should support specifying this as an option in BigQueryIO.
Imported from Jira [BEAM-3167](https://issues.apache.org/jira/browse/BEAM-3167). Original Jira may contain additional context.
Reported by: chamikara.
Contributor guide
Research direction
Start in sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java at the referenced load-job configuration. Read the BigQuery load-job schema autodetection documentation, then trace how BigQueryIO options reach that configuration; done means callers can specify schema autodetection and load jobs receive the requested setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- cloud, data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100