[Improvement]: Create a job manager for Python BigQueryIO
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
Java's BigQuery connector delegates waiting for jobs to a [PendingJobManager](https://github.com/apache/beam/blob/935eeadd45a6100c68243fc5b266b20d26164173/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java#L91). The benefit with this job manager is that jobs are waited for and their state is checked on in parallel. This way, we get [useful logs](https://github.com/apache/beam/blob/935eeadd45a6100c68243fc5b266b20d26164173/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java#L135) and can determine if a job is significantly slower than the rest.
In contrast, the Python connector waits for jobs and checks on their state sequentially. The connector could benefit from having a job manager.
### Issue Priority
Priority: 3
### Issue Component
Component: io-py-gcp
Contributor guide
Assessment
This issue has not been assessed yet.