googleapis / googleapis/python-genai
google.auth.exceptions.TransportError is not retried despite HttpRetryOptions
- Dominant language
- Python
- Stars
- 4k
- Forks
- 1k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 40
Description
### Is this a client library issue or a product issue?
This is a `google-genai` client library issue. A transient Google Auth transport
failure bypasses the retry policy even when `HttpRetryOptions` is explicitly
configured.
### Environment
- Python: 3.12
- Reproduced with `google-genai==2.12.1` and `google-auth==2.55.2`
- Confirmed against `google-genai==2.18.1`, `google-auth==2.56.0`, and current `main`
- Vertex AI with service-account Application Default Credentials
### Description
When Vertex credentials need refreshing, `google-genai` calls
`credentials.refresh()` before dispatching the model request. Network failures
from that refresh are wrapped by Google Auth as
`google.auth.exceptions.TransportError`.
The complete `_async_request_once()` operation, including credential refresh,
is already wrapped by the configured Tenacity policy. However, `retry_args()`
only retries selected `APIError` status codes and HTTPX timeout/connect errors.
It does not include Google Auth's equivalent transport exception, so the first
refresh failure terminates the request regardless of the configured attempt
count.
### Network-free reproduction
```python
import httpx
import tenacity
from google.auth.exceptions import TransportError
from google.genai._api_client import retry_args
from google.genai.types import HttpRetryOptions
def attempts_for(error):
attempts = 0
def fail():
nonlocal attempts
attempts += 1
raise error
retrying = tenacity.Retrying(
**retry_args(
HttpRetryOptions(
attempts=3,
initial_delay=0.001,
max_delay=0.001,
jitter=0.001,
)
)
)
try:
retrying(fail)
except Exception:
pass
return attempts
print(attempts_for(httpx.ConnectError('synthetic')))
print(attempts_for(TransportError('synthetic')))
```
Actual output:
```text
3
1
```
Expected output:
```text
3
3
```
An end-to-end reproduction with flaky credentials and a mocked model transport
also produces one credential refresh attempt, zero model HTTP requests, and an
immediately propagated `TransportError`. After including `TransportError` in
the predicate, the second credential refresh succeeds and exactly one model
HTTP request is dispatched.
### Root cause
The current predicate is:
```python
lambda e: (isinstance(e, errors.APIError) and e.code in retriable_codes)
or isinstance(e, _HTTPX_TRANSIENT_EXC)
```
`google.auth.exceptions.TransportError` matches neither branch.
### Prior art
- #2337 reported the equivalent omission for HTTPX transport exceptions.
- #2345 added `httpx.TimeoutException` and `httpx.ConnectError` to this predicate.
- googleapis/google-auth-library-python#737 concluded that client-library retry
predicates should account for `google.auth.exceptions.TransportError`.
- Google Cloud Storage and BigQuery include `TransportError` in their retry
predicates.
### Suggested fix
Treat `google.auth.exceptions.TransportError` as retryable when
`HttpRetryOptions` is enabled, while continuing to exclude permanent
authentication failures such as `RefreshError`.
The aiohttp request branches currently perform one separate hard-coded retry
for `TransportError`. If the shared Tenacity predicate is extended, that path
may eventually benefit from consolidation to avoid nested retry policies. That
existing aiohttp behaviour is separate from the credential-refresh failure
above, which occurs before model dispatch.
Contributor guide
Assessment
This issue has not been assessed yet.