googleapis / googleapis/python-genai

google.auth.exceptions.TransportError is not retried despite HttpRetryOptions

Open
#2,869 2 comments 0 reactions 1 assignee Assigned to @kkorpal View on GitHub
priority: p2 type: bug
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

### Is this a client library issue or a product issue?

This is a `google-genai` client library issue. A transient Google Auth transport
failure bypasses the retry policy even when `HttpRetryOptions` is explicitly
configured.

### Environment

- Python: 3.12
- Reproduced with `google-genai==2.12.1` and `google-auth==2.55.2`
- Confirmed against `google-genai==2.18.1`, `google-auth==2.56.0`, and current `main`
- Vertex AI with service-account Application Default Credentials

### Description

When Vertex credentials need refreshing, `google-genai` calls
`credentials.refresh()` before dispatching the model request. Network failures
from that refresh are wrapped by Google Auth as
`google.auth.exceptions.TransportError`.

The complete `_async_request_once()` operation, including credential refresh,
is already wrapped by the configured Tenacity policy. However, `retry_args()`
only retries selected `APIError` status codes and HTTPX timeout/connect errors.
It does not include Google Auth's equivalent transport exception, so the first
refresh failure terminates the request regardless of the configured attempt
count.

### Network-free reproduction

```python
import httpx
import tenacity

from google.auth.exceptions import TransportError
from google.genai._api_client import retry_args
from google.genai.types import HttpRetryOptions

def attempts_for(error):
attempts = 0

def fail():
nonlocal attempts
attempts += 1
raise error

retrying = tenacity.Retrying(
**retry_args(
HttpRetryOptions(
attempts=3,
initial_delay=0.001,
max_delay=0.001,
jitter=0.001,
)
)
)
try:
retrying(fail)
except Exception:
pass
return attempts

print(attempts_for(httpx.ConnectError('synthetic')))
print(attempts_for(TransportError('synthetic')))
```

Actual output:

```text
3
1
```

Expected output:

```text
3
3
```

An end-to-end reproduction with flaky credentials and a mocked model transport
also produces one credential refresh attempt, zero model HTTP requests, and an
immediately propagated `TransportError`. After including `TransportError` in
the predicate, the second credential refresh succeeds and exactly one model
HTTP request is dispatched.

### Root cause

The current predicate is:

```python
lambda e: (isinstance(e, errors.APIError) and e.code in retriable_codes)
or isinstance(e, _HTTPX_TRANSIENT_EXC)
```

`google.auth.exceptions.TransportError` matches neither branch.

### Prior art

- #2337 reported the equivalent omission for HTTPX transport exceptions.
- #2345 added `httpx.TimeoutException` and `httpx.ConnectError` to this predicate.
- googleapis/google-auth-library-python#737 concluded that client-library retry
predicates should account for `google.auth.exceptions.TransportError`.
- Google Cloud Storage and BigQuery include `TransportError` in their retry
predicates.

### Suggested fix

Treat `google.auth.exceptions.TransportError` as retryable when
`HttpRetryOptions` is enabled, while continuing to exclude permanent
authentication failures such as `RefreshError`.

The aiohttp request branches currently perform one separate hard-coded retry
for `TransportError`. If the shared Tenacity predicate is extended, that path
may eventually benefit from consolidation to avoid nested retry policies. That
existing aiohttp behaviour is separate from the credential-refresh failure
above, which occurs before model dispatch.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.