googleapis / googleapis/python-genai

Async file upload retry blocks the event loop

Open
#2,938 1 comment 0 reactions 1 assignee Claimed by @Venkaiahbabuneelam View on GitHub
priority: p2 type: bug
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

#### Environment details

- Programming language: Python
- OS: macOS
- Language runtime version: 3.12.14
- Package version: 2.22.0 and `main` at `0ec3d8a4b2c85817434045dad739f6227c2d5c4c`

#### Steps to reproduce

1. Start a local HTTP server that returns `200` without `x-goog-upload-status` for the first resumable upload request, then returns `final` on the second request.
2. Call the HTTPX path used by `client.aio.files.upload` while a second coroutine sleeps for 50 ms.
3. Measure when the second coroutine resumes.

The local server avoids external API and quota dependencies. The upload retry is the real SDK and HTTPX path.

Reproducer

```python
import asyncio
import io
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

from google.genai import _api_client, types

class Handler(BaseHTTPRequestHandler):
attempts = 0

def do_POST(self):
type(self).attempts += 1
self.rfile.read(int(self.headers['Content-Length']))
self.send_response(200)
if type(self).attempts == 2:
self.send_header('X-Goog-Upload-Status', 'final')
self.end_headers()

def log_message(self, format, *args):
pass

async def main():
server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
base_url = f'http://127.0.0.1:{server.server_port}'
client = _api_client.BaseApiClient(
vertexai=False,
api_key='test',
http_options=types.HttpOptions(
base_url=base_url,
async_client_args={'trust_env': False},
),
)
client._use_aiohttp = lambda: False
_api_client.INITIAL_RETRY_DELAY = 0.25
started = time.monotonic()

async def ticker():
await asyncio.sleep(0.05)
return time.monotonic() - started

try:
_, tick = await asyncio.gather(
client._async_upload_fd(
io.BytesIO(b'test'), f'{base_url}/upload', 4
),
ticker(),
)
print(f'attempts={Handler.attempts}')
print(f'ticker_delay={tick:.3f}s')
print(f'event_loop_blocked={tick >= 0.20}')
finally:
await client.aclose()
server.shutdown()
server.server_close()

asyncio.run(main())
```

```text
attempts=2
ticker_delay=0.281s
event_loop_blocked=True
```

Cancellation is delayed by the same backoff:

```text
attempts=1
cancel_requested_after=0.433s
cancelled_after=0.434s
backoff_blocked_cancellation=True
```

The HTTPX branch in `_async_upload_fd` calls `time.sleep` during retry backoff. This blocks the event loop, so unrelated coroutines and cancellation cannot run until the sleep returns. The aiohttp branch already uses `await asyncio.sleep`.

Expected behavior: async upload retry backoff should yield to the event loop and remain cancellable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.