googleapis / googleapis/google-cloud-python

bigquery: SSLError marked globally non-retryable (#17489) fails jobs.get / result() polling on transient TLS resets

Abierto
#18,178 1 comentario 0 reacciones 1 asignado Reclamado por @daniel-sanche Ver en GitHub
api: bigquery priority: p2 type: bug
Lenguaje dominante
Python
Estrellas
5.4k
Forks
1.8k
Merge medio
3 d 4 h
PR fusionados (30 d)
122

Descripción

### Determine this is the right repository

- [x] I determined this is the correct repository in which to report this bug.

### Summary of the issue

[#17489](https://github.com/googleapis/google-cloud-python/pull/17489) ("avoid SSLError retry loop", released in 3.42.2) added `requests.exceptions.SSLError` to a new `_UNSTRUCTURED_NON_RETRYABLE_TYPES` and short-circuits `_should_retry` on it
([`retry.py:46,76-77`](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-bigquery/google/cloud/bigquery/retry.py#L46)).

The PR's motivating case is `insert_rows_json`, but `_should_retry` is not that method's predicate — it is the base predicate for every retry surface in the client. As a result **a single transient TLS reset now fails any API call**, where 3.40.1 retried it as the `requests.exceptions.ConnectionError` subclass it is.

#### Expected vs. actual

A transient `SSLEOFError` (peer or middlebox dropping the socket mid-handshake) is a transport error and was retried to `DEFAULT_RETRY`'s 600 s deadline in 3.40.1 — `retry.py` there has no `_UNSTRUCTURED_NON_RETRYABLE_TYPES` at all, so `_should_retry` fell through to the `isinstance` check against `_UNSTRUCTURED_RETRYABLE_TYPES`, which contains `requests.exceptions.ConnectionError` (and `requests` makes `SSLError` a subclass of it deliberately). In 3.42.2 the same blip raises on the first occurrence.

### API client name and version

google-cloud-bigquery: 3.42.2 (first release with the change) — still present in 3.43.0

### Reproduction steps: code

_No response_

### Reproduction steps: supporting files

_No response_

### Reproduction steps: actual results

_No response_

### Reproduction steps: expected results

_No response_

### OS & version + platform

Container-Optimized OS (GKE, Cloud Composer 3)

### Python environment

Python: 3.11

### Python dependencies

_No response_

### Additional context

#### What we actually observe in production

Deferred `BigQueryInsertJobOperator` tasks (Airflow 3.2 / Composer 3) poll `jobs.get` every 4 s. Since upgrading to an image carrying 3.42.2, polls fail like this — and each failure fails the task and re-submits the query, on jobs that are `RUNNING` and healthy:

```
HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects//jobs/?projection=full&location=EU (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol')))
```

49 such events across ~20 different DAGs in ~19 h; zero on the same workload while it ran 3.40.1. Note the URL family: `jobs.get` — a GET with no request body, so no schema can be involved. Note also that `google.auth.transport.requests.AuthorizedSession` mounts `HTTPAdapter(max_retries=3)` on the **auth** session only, so nothing below the client retries the data request either: identical resets on an IAM refresh call recover silently, while `jobs.get` fails outright.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.