googleapis / googleapis/google-cloud-python

bigquery: SSLError marked globally non-retryable (#17489) fails jobs.get / result() polling on transient TLS resets

未关闭
#18,178 1 条评论 0 个 reaction 已指派 1 人 已被 @daniel-sanche 认领 在 GitHub 查看
api: bigquery priority: p2 type: bug
主要语言
Python
星标
5.4k
派生
1.8k
平均合并
3 天 4 小时
30 天内合并 PR
122

描述

### Determine this is the right repository

- [x] I determined this is the correct repository in which to report this bug.

### Summary of the issue

[#17489](https://github.com/googleapis/google-cloud-python/pull/17489) ("avoid SSLError retry loop", released in 3.42.2) added `requests.exceptions.SSLError` to a new `_UNSTRUCTURED_NON_RETRYABLE_TYPES` and short-circuits `_should_retry` on it
([`retry.py:46,76-77`](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-bigquery/google/cloud/bigquery/retry.py#L46)).

The PR's motivating case is `insert_rows_json`, but `_should_retry` is not that method's predicate — it is the base predicate for every retry surface in the client. As a result **a single transient TLS reset now fails any API call**, where 3.40.1 retried it as the `requests.exceptions.ConnectionError` subclass it is.

#### Expected vs. actual

A transient `SSLEOFError` (peer or middlebox dropping the socket mid-handshake) is a transport error and was retried to `DEFAULT_RETRY`'s 600 s deadline in 3.40.1 — `retry.py` there has no `_UNSTRUCTURED_NON_RETRYABLE_TYPES` at all, so `_should_retry` fell through to the `isinstance` check against `_UNSTRUCTURED_RETRYABLE_TYPES`, which contains `requests.exceptions.ConnectionError` (and `requests` makes `SSLError` a subclass of it deliberately). In 3.42.2 the same blip raises on the first occurrence.

### API client name and version

google-cloud-bigquery: 3.42.2 (first release with the change) — still present in 3.43.0

### Reproduction steps: code

_No response_

### Reproduction steps: supporting files

_No response_

### Reproduction steps: actual results

_No response_

### Reproduction steps: expected results

_No response_

### OS & version + platform

Container-Optimized OS (GKE, Cloud Composer 3)

### Python environment

Python: 3.11

### Python dependencies

_No response_

### Additional context

#### What we actually observe in production

Deferred `BigQueryInsertJobOperator` tasks (Airflow 3.2 / Composer 3) poll `jobs.get` every 4 s. Since upgrading to an image carrying 3.42.2, polls fail like this — and each failure fails the task and re-submits the query, on jobs that are `RUNNING` and healthy:

```
HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects//jobs/?projection=full&location=EU (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol')))
```

49 such events across ~20 different DAGs in ~19 h; zero on the same workload while it ran 3.40.1. Note the URL family: `jobs.get` — a GET with no request body, so no schema can be involved. Note also that `google.auth.transport.requests.AuthorizedSession` mounts `HTTPAdapter(max_retries=3)` on the **auth** session only, so nothing below the client retries the data request either: identical resets on an IAM refresh call recover silently, while `jobs.get` fails outright.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。