[Feature Request]: Cache the BigQuery table definition instead of calling tables.get() from every worker
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
When calling BigQuery's tables.get method, today, we are calling it from every Dataflow worker. This is resulting in customers hitting Quota issues. Please can we cache the Table definition and re-use for all workers instead of calling the method from every worker.
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/get
https://github.com/apache/beam/blob/9a1e7bf4cd399177e5de7fe28bad741cade452a5/sdks/python/apache_beam/io/gcp/bigquery.py#L902-L903
https://github.com/apache/beam/blob/9a1e7bf4cd399177e5de7fe28bad741cade452a5/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L565
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.