[Feature Request]: Support SchemaUpdateOptions for BigQueryIO Storage Write API
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
When writing data with BigQueryIO using `BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE` or `BigQueryIO.Write.Method.STORAGE_WRITE_API`, it would be very useful to be able to specify one or more `SchemaUpdateOption` values via `.withSchemaUpdateOptions`.
Currently this only works for `BigQueryIO.Write.Method.FILE_LOADS`, which means that for other insert methods new fields are either ignored (if `ignoreUnknownValues` is set), or fail to insert. In our use case, we are streaming Avro records from Kafka to BigQuery, with the schema constantly changing (mostly new fields). Having BigQueryIO manage this would be a huge benefit.
### Issue Priority
Priority: 3
### Issue Component
Component: io-java-gcp
Contributor guide
Assessment
This issue has not been assessed yet.