apache / apache/beam

[Feature Request]: Support SchemaUpdateOptions for BigQueryIO Storage Write API

Open
#24,063 2 comments 0 reactions 0 assignees View on GitHub
bigquery gcp io java new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

When writing data with BigQueryIO using `BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE` or `BigQueryIO.Write.Method.STORAGE_WRITE_API`, it would be very useful to be able to specify one or more `SchemaUpdateOption` values via `.withSchemaUpdateOptions`.

Currently this only works for `BigQueryIO.Write.Method.FILE_LOADS`, which means that for other insert methods new fields are either ignored (if `ignoreUnknownValues` is set), or fail to insert. In our use case, we are streaming Avro records from Kafka to BigQuery, with the schema constantly changing (mostly new fields). Having BigQueryIO manage this would be a huge benefit.

### Issue Priority

Priority: 3

### Issue Component

Component: io-java-gcp

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.