googleapis / googleapis/google-cloud-python

bigquery: SSLError marked globally non-retryable (#17489) fails jobs.get / result() polling on transient TLS resets

Đang mở
#18,178 1 bình luận 0 reaction 1 người được giao Được @daniel-sanche nhận Xem trên GitHub
api: bigquery priority: p2 type: bug
Ngôn ngữ chính
Python
Star
5.4k
Fork
1.8k
Merge trung bình
3 ngày 4 giờ
Pull request đã merge (30 ngày)
122

Mô tả

### Determine this is the right repository

- [x] I determined this is the correct repository in which to report this bug.

### Summary of the issue

[#17489](https://github.com/googleapis/google-cloud-python/pull/17489) ("avoid SSLError retry loop", released in 3.42.2) added `requests.exceptions.SSLError` to a new `_UNSTRUCTURED_NON_RETRYABLE_TYPES` and short-circuits `_should_retry` on it
([`retry.py:46,76-77`](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-bigquery/google/cloud/bigquery/retry.py#L46)).

The PR's motivating case is `insert_rows_json`, but `_should_retry` is not that method's predicate — it is the base predicate for every retry surface in the client. As a result **a single transient TLS reset now fails any API call**, where 3.40.1 retried it as the `requests.exceptions.ConnectionError` subclass it is.

#### Expected vs. actual

A transient `SSLEOFError` (peer or middlebox dropping the socket mid-handshake) is a transport error and was retried to `DEFAULT_RETRY`'s 600 s deadline in 3.40.1 — `retry.py` there has no `_UNSTRUCTURED_NON_RETRYABLE_TYPES` at all, so `_should_retry` fell through to the `isinstance` check against `_UNSTRUCTURED_RETRYABLE_TYPES`, which contains `requests.exceptions.ConnectionError` (and `requests` makes `SSLError` a subclass of it deliberately). In 3.42.2 the same blip raises on the first occurrence.

### API client name and version

google-cloud-bigquery: 3.42.2 (first release with the change) — still present in 3.43.0

### Reproduction steps: code

_No response_

### Reproduction steps: supporting files

_No response_

### Reproduction steps: actual results

_No response_

### Reproduction steps: expected results

_No response_

### OS & version + platform

Container-Optimized OS (GKE, Cloud Composer 3)

### Python environment

Python: 3.11

### Python dependencies

_No response_

### Additional context

#### What we actually observe in production

Deferred `BigQueryInsertJobOperator` tasks (Airflow 3.2 / Composer 3) poll `jobs.get` every 4 s. Since upgrading to an image carrying 3.42.2, polls fail like this — and each failure fails the task and re-submits the query, on jobs that are `RUNNING` and healthy:

```
HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects//jobs/?projection=full&location=EU (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol')))
```

49 such events across ~20 different DAGs in ~19 h; zero on the same workload while it ran 3.40.1. Note the URL family: `jobs.get` — a GET with no request body, so no schema can be involved. Note also that `google.auth.transport.requests.AuthorizedSession` mounts `HTTPAdapter(max_retries=3)` on the **auth** session only, so nothing below the client retries the data request either: identical resets on an IAM refresh call recover silently, while `jobs.get` fails outright.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.