GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

Consider retry in BigQuery connect code

Open
#1,313 0 comments 0 reactions 0 assignees View on GitHub
priority: p2 type: testing
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

When running integration tests we occasionally have failures due to transient errors connecting to BigQuery:

Example:
```
"Step #13 - "integration_db2": data_validation/clients.py:282: "
"Step #13 - "integration_db2": data_validation/clients.py:105: in get_bigquery_client"
"Step #13 - "integration_db2": ibis_client = ibis.bigquery.connect("
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/ibis/__init__.py:97: in connect"
"Step #13 - "integration_db2": return backend.connect(*args, **kwargs)"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/ibis/backends/base/__init__.py:530: in connect"
"Step #13 - "integration_db2": new_backend.reconnect()"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/ibis/backends/base/__init__.py:545: in reconnect"
"Step #13 - "integration_db2": self.do_connect(*self._con_args, **self._con_kwargs)"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/ibis/backends/bigquery/__init__.py:163: in do_connect"
"Step #13 - "integration_db2": credentials, default_project_id = pydata_google_auth.default("
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/pydata_google_auth/auth.py:152: in default"
"Step #13 - "integration_db2": credentials = get_user_credentials("
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/pydata_google_auth/auth.py:362: in get_user_credentials"
"Step #13 - "integration_db2": credentials = _webserver.run_local_server(app_flow, **AUTH_URI_KWARGS)"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/pydata_google_auth/_webserver.py:89: in run_local_server"
"Step #13 - "integration_db2": return app_flow.run_local_server(host=LOCALHOST, port=port, **kwargs)"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/google_auth_oauthlib/flow.py:447: in run_local_server"
"Step #13 - "integration_db2": webbrowser.get(browser).open(auth_url, new=1, autoraise=True)"
...
"Step #13 - "integration_db2": > raise Error("could not locate runnable browser")"
"Step #13 - "integration_db2": E webbrowser.Error: could not locate runnable browser"
"Step #13 - "integration_db2": /usr/local/lib/python3.9/webbrowser.py:65: Error"
```

While transient errors during tests is only a small irritation we can probably assume that intense users of DVT on BigQuery could also run into these errors.

We could consider adding a single retry in `get_bigquery_client()` in `data_validation/clients.py`.

We could investigate `from google.api_core import retry` as a property to auto retry (a single time) `get_bigquery_client()` if it throws with `webbrowser.Error`.

I've not fully researched all retry attributes but I believe this would do the trick in a clean way.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.