apache / apache/beam

[Feature Request]: WriteToBigquery should offer a callback to deeply customise destination & element to BQ

Open
#23,746 2 comments 0 reactions 0 assignees View on GitHub
gcp io new feature P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Hi, Beam community,

I very much like the feature WriteToBigquery offering to dynamically deciding the `table`:

https://github.com/apache/beam/blob/6adecd438790d8c1b5182043db16232b68ff7a98/sdks/python/apache_beam/io/gcp/bigquery.py#L1788

I am thinking about a potential improvement opportunity. Currently `WriteToBigquery` supports either dict or TableRow, and it assumes that table reference can be inferred from the content of dict or TableRow to be written into BQ. We probably should remove that assumption. For systems that does not align with this assumption the operator would force them to add table name as a column. (or optionally enable `ignore_unknown_fields`)

This is a case I am expecting WriteToBigquery to support:

Given a PCollection of `K, V` pair of , I hope `K` can be the used to infer table name yet get `V` written into the dictionary. (or even do some further processing on V)

I can see there is a few way to do that. The simplest way I can think of is to allow `table` arg to be an instance of `DoFn` or `AppendDestinationsFn`.

https://github.com/apache/beam/blob/b5a0f4895ccf7bd2f8b03217d866e8d0c67f6f52/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L1488

If the sdk allow us to overwrite `AppendDestinationsFn` and pass it into `WriteToBigquery` we will have more freedom to customise the class.

I am happy to contribute and make this feature come true. But would be keen to get communities' feedback first:)

### Issue Priority

Priority: 2

### Issue Component

Component: io-py-gcp

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.