googleapis / googleapis/python-genai

heavy calls die behind NAT idle-eviction because the default httpx transport doesn't set any TCP keepalive options

Open
#2,705 2 comments 0 reactions 1 assignee Claimed by @Venkaiahbabuneelam View on GitHub
priority: p2 type: bug
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

### Summary

`generate_content` can hang indefinitely (or raise a `ReadTimeout`) when a call runs longer than the TCP idle timeout of a NAT Gateway / firewall on the egress path. The default transport sets no TCP keepalive, so a connection that goes silent while awaiting the response gets dropped.

With no timeout set the read hangs forever; with one set it raises `ReadTimeout` at the deadline.

Likely the same root cause as #1893. For reference, the Anthropic Python SDK enables keepalive by default (`SO_KEEPALIVE` + `TCP_KEEPIDLE=60`), so it isn't affected: [`_base_client.py`](https://github.com/anthropics/anthropic-sdk-python/blob/main/src/anthropic/_base_client.py#L906-L930).

### Environment details

- Programming language: Python
- OS: Linux (container)
- Language runtime version: CPython 3.14
- Package version: google-genai 1.65.0

### Steps to reproduce

1. Run behind a NAT Gateway / stateful firewall with a short idle timeout (e.g. ~3-4 min).
2. Make a heavy `generate_content` call that stays silent for longer than that idle timeout.
3. With no read timeout: the call hangs forever. With one set: it raises a `ReadTimeout` at the deadline.
4. Enable TCP keepalive on a custom httpx transport (see below) & the issue disappears.

### Workaround

```python
import socket
import httpx
from google import genai
from google.genai import types

# Linux-specific socket options
_KEEPALIVE = [
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60), # first probe after 60s idle
(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 15),
(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 4),
]
transport = httpx.HTTPTransport(socket_options=_KEEPALIVE)
client = genai.Client( # add vertexai=True, project=..., location=... for Vertex
http_options=types.HttpOptions(httpx_client=httpx.Client(transport=transport)),
)
```

Related: #1893

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.