apache / apache/beam

[Improvement]: Create a job manager for Python BigQueryIO

Open
#23,235 1 comment 0 reactions 0 assignees View on GitHub
gcp improvement io P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Java's BigQuery connector delegates waiting for jobs to a [PendingJobManager](https://github.com/apache/beam/blob/935eeadd45a6100c68243fc5b266b20d26164173/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java#L91). The benefit with this job manager is that jobs are waited for and their state is checked on in parallel. This way, we get [useful logs](https://github.com/apache/beam/blob/935eeadd45a6100c68243fc5b266b20d26164173/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java#L135) and can determine if a job is significantly slower than the rest.

In contrast, the Python connector waits for jobs and checks on their state sequentially. The connector could benefit from having a job manager.

### Issue Priority

Priority: 3

### Issue Component

Component: io-py-gcp

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.