GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
Consider retry in BigQuery connect code
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
When running integration tests we occasionally have failures due to transient errors connecting to BigQuery:
Example:
```
"Step #13 - "integration_db2": data_validation/clients.py:282: "
"Step #13 - "integration_db2": data_validation/clients.py:105: in get_bigquery_client"
"Step #13 - "integration_db2": ibis_client = ibis.bigquery.connect("
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/ibis/__init__.py:97: in connect"
"Step #13 - "integration_db2": return backend.connect(*args, **kwargs)"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/ibis/backends/base/__init__.py:530: in connect"
"Step #13 - "integration_db2": new_backend.reconnect()"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/ibis/backends/base/__init__.py:545: in reconnect"
"Step #13 - "integration_db2": self.do_connect(*self._con_args, **self._con_kwargs)"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/ibis/backends/bigquery/__init__.py:163: in do_connect"
"Step #13 - "integration_db2": credentials, default_project_id = pydata_google_auth.default("
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/pydata_google_auth/auth.py:152: in default"
"Step #13 - "integration_db2": credentials = get_user_credentials("
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/pydata_google_auth/auth.py:362: in get_user_credentials"
"Step #13 - "integration_db2": credentials = _webserver.run_local_server(app_flow, **AUTH_URI_KWARGS)"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/pydata_google_auth/_webserver.py:89: in run_local_server"
"Step #13 - "integration_db2": return app_flow.run_local_server(host=LOCALHOST, port=port, **kwargs)"
"Step #13 - "integration_db2": .nox/integration_db2/lib/python3.9/site-packages/google_auth_oauthlib/flow.py:447: in run_local_server"
"Step #13 - "integration_db2": webbrowser.get(browser).open(auth_url, new=1, autoraise=True)"
...
"Step #13 - "integration_db2": > raise Error("could not locate runnable browser")"
"Step #13 - "integration_db2": E webbrowser.Error: could not locate runnable browser"
"Step #13 - "integration_db2": /usr/local/lib/python3.9/webbrowser.py:65: Error"
```
While transient errors during tests is only a small irritation we can probably assume that intense users of DVT on BigQuery could also run into these errors.
We could consider adding a single retry in `get_bigquery_client()` in `data_validation/clients.py`.
We could investigate `from google.api_core import retry` as a property to auto retry (a single time) `get_bigquery_client()` if it throws with `webbrowser.Error`.
I've not fully researched all retry attributes but I believe this would do the trick in a clean way.
Contributor guide
Assessment
This issue has not been assessed yet.