BigQuery Streaming Inserts in Python perform GetTable call too many times
Open
gcp
improvement
io
P3
python
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Create table if needed is called once per destination per bundle. We should call it only once per destination at all. This can be done by passing the created tables as a side input.
[https://github.com/apache/beam/blob/v2.13.0/sdks/python/apache_beam/io/gcp/bigquery.py#L804-L816](https://github.com/apache/beam/blob/v2.13.0/sdks/python/apache_beam/io/gcp/bigquery.py#L804-L816)
Imported from Jira [BEAM-7803](https://issues.apache.org/jira/browse/BEAM-7803). Original Jira may contain additional context.
Reported by: pabloem.
Contributor guide
Assessment
This issue has not been assessed yet.