apache / apache/beam

Long BigQuery dry runs cause avalanche delay

Open
#19,970 0 comments 4 reactions 0 assignees View on GitHub
bug gcp io java P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Reproduction Steps:

1. Compose a BigQuery SELECT query that will take over 80 seconds for a dry run.
2. Run the query with Beam SDK's BigQueryIO.
3. Observe the 10**** minute delay before the actual query job is created.

When running readTableRows(), BigQueryIO attempts to estimate the query size by performing a dry run, even if withoutValidation() is set. If the request takes over 80 seconds (RetryHttpRequestInitializer.HANGING_GET_TIMEOUT_SEC), RetryHttpRequestInitializer will time out and retry, up to 9 times (BigQueryServicesImpl.MAX_RPC_RETRIES). Hence, once a dry run duration crosses the 80 second tipping point, it causes an inevitable avalanche of a 720-second delay. Considering the fact that size estimation is not a requirement in running the query [1], BigQueryIO should provide a way to circumvent the redundant delay, especially in consideration of time-critical enterprise workloads.

There can be several ways to address this:
- increasing the timeout threshold (which will still create a tipping point);
- preventing the dry run requests from retrying; or
- adding an option to skip the size estimation within serializeToCloudSource().

[1] https://github.com/apache/beam/blob/2ec3b0495c191597c9a88830d25a2c360b3277e0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/CustomSources.java#L75

Imported from Jira [BEAM-8906](https://issues.apache.org/jira/browse/BEAM-8906). Original Jira may contain additional context.
Reported by: juneoh.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.