googleapis / googleapis/google-cloud-python
bigquery: SSLError marked globally non-retryable (#17489) fails jobs.get / result() polling on transient TLS resets
- Vorherrschende Sprache
- Python
- Sterne
- 5.4k
- Forks
- 1.8k
- Ø Merge
- 3 T. 4 Std.
- Gemergte PRs (30 T.)
- 122
Beschreibung
### Determine this is the right repository
- [x] I determined this is the correct repository in which to report this bug.
### Summary of the issue
[#17489](https://github.com/googleapis/google-cloud-python/pull/17489) ("avoid SSLError retry loop", released in 3.42.2) added `requests.exceptions.SSLError` to a new `_UNSTRUCTURED_NON_RETRYABLE_TYPES` and short-circuits `_should_retry` on it
([`retry.py:46,76-77`](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-bigquery/google/cloud/bigquery/retry.py#L46)).
The PR's motivating case is `insert_rows_json`, but `_should_retry` is not that method's predicate — it is the base predicate for every retry surface in the client. As a result **a single transient TLS reset now fails any API call**, where 3.40.1 retried it as the `requests.exceptions.ConnectionError` subclass it is.
#### Expected vs. actual
A transient `SSLEOFError` (peer or middlebox dropping the socket mid-handshake) is a transport error and was retried to `DEFAULT_RETRY`'s 600 s deadline in 3.40.1 — `retry.py` there has no `_UNSTRUCTURED_NON_RETRYABLE_TYPES` at all, so `_should_retry` fell through to the `isinstance` check against `_UNSTRUCTURED_RETRYABLE_TYPES`, which contains `requests.exceptions.ConnectionError` (and `requests` makes `SSLError` a subclass of it deliberately). In 3.42.2 the same blip raises on the first occurrence.
### API client name and version
google-cloud-bigquery: 3.42.2 (first release with the change) — still present in 3.43.0
### Reproduction steps: code
_No response_
### Reproduction steps: supporting files
_No response_
### Reproduction steps: actual results
_No response_
### Reproduction steps: expected results
_No response_
### OS & version + platform
Container-Optimized OS (GKE, Cloud Composer 3)
### Python environment
Python: 3.11
### Python dependencies
_No response_
### Additional context
#### What we actually observe in production
Deferred `BigQueryInsertJobOperator` tasks (Airflow 3.2 / Composer 3) poll `jobs.get` every 4 s. Since upgrading to an image carrying 3.42.2, polls fail like this — and each failure fails the task and re-submits the query, on jobs that are `RUNNING` and healthy:
```
HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects//jobs/?projection=full&location=EU (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol')))
```
49 such events across ~20 different DAGs in ~19 h; zero on the same workload while it ran 3.40.1. Note the URL family: `jobs.get` — a GET with no request body, so no schema can be involved. Note also that `google.auth.transport.requests.AuthorizedSession` mounts `HTTPAdapter(max_retries=3)` on the **auth** session only, so nothing below the client retries the data request either: identical resets on an IAM refresh call recover silently, while `jobs.get` fails outright.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.