[Feature Request]: WriteToBigquery should offer a callback to deeply customise destination & element to BQ
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
Hi, Beam community,
I very much like the feature WriteToBigquery offering to dynamically deciding the `table`:
https://github.com/apache/beam/blob/6adecd438790d8c1b5182043db16232b68ff7a98/sdks/python/apache_beam/io/gcp/bigquery.py#L1788
I am thinking about a potential improvement opportunity. Currently `WriteToBigquery` supports either dict or TableRow, and it assumes that table reference can be inferred from the content of dict or TableRow to be written into BQ. We probably should remove that assumption. For systems that does not align with this assumption the operator would force them to add table name as a column. (or optionally enable `ignore_unknown_fields`)
This is a case I am expecting WriteToBigquery to support:
Given a PCollection of `K, V` pair of , I hope `K` can be the used to infer table name yet get `V` written into the dictionary. (or even do some further processing on V)
I can see there is a few way to do that. The simplest way I can think of is to allow `table` arg to be an instance of `DoFn` or `AppendDestinationsFn`.
https://github.com/apache/beam/blob/b5a0f4895ccf7bd2f8b03217d866e8d0c67f6f52/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L1488
If the sdk allow us to overwrite `AppendDestinationsFn` and pass it into `WriteToBigquery` we will have more freedom to customise the class.
I am happy to contribute and make this feature come true. But would be keen to get communities' feedback first:)
### Issue Priority
Priority: 2
### Issue Component
Component: io-py-gcp
Contributor guide
Assessment
This issue has not been assessed yet.