lablup / lablup/backend.ai

VFolder.download and Request.connect_websocket bypass APIConfig timeouts

Open
#11,403 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Summary

Two paths in the Backend.AI Python client (`backend.ai-client==26.4.4rc2`) construct their own HTTP/WS client objects for outbound calls without propagating `APIConfig.connection_timeout` / `APIConfig.read_timeout`. A wedged Manager or storage backend therefore hangs the caller indefinitely, defeating the contract the rest of the SDK honors via `Request.fetch()` (which builds a per-call `aiohttp.ClientTimeout(sock_connect=…, sock_read=…)`).

Surfaced during the SDK timeout audit on lablup/backend.ai-fasttrack#3774 (PR lablup/backend.ai-fasttrack#3776).

## Affected paths

### 1. `Request.connect_websocket` — `request.py:347-386`

The websocket-upgrade path constructs the upgrade request without an `aiohttp.ClientTimeout`, relying on `aiohttp` defaults. A wedged Manager during the upgrade handshake will hang indefinitely.

### 2. `VFolder.download()` — `func/vfolder.py:364`

The direct-to-storage transfer constructs a fresh `aiohttp.ClientSession()` without propagating `APIConfig` timeouts. A hung storage backend will block the call indefinitely.

## Downstream impact (FastTrack)

- `workflow/experiments/metrics_ingestion.py` calls `vfolder.download(...)` from an outbox handler. A wedged storage backend hangs an outbox worker; the pool-limited workers back up the entire experiment-metrics ingestion queue until killed.
- FastTrack has shipped a thread-pool-deadline workaround in lablup/backend.ai-fasttrack#3779 to free outbox workers, but the worker thread still leaks because the SDK exposes no cancellation hook.
- No FastTrack call sites use `connect_websocket` today (informational only) — but it's a real risk for any future Manager-WS consumer.

## Suggested fix

Have both paths build an `aiohttp.ClientTimeout(sock_connect=APIConfig.connection_timeout, sock_read=APIConfig.read_timeout)` from the active config and pass it through, exactly as `Request.fetch()` already does. This restores the SDK-wide invariant that every outbound call honors the user's `APIConfig` deadline.

## References

- Audit issue: lablup/backend.ai-fasttrack#3774
- Audit PR: lablup/backend.ai-fasttrack#3776
- FastTrack workaround: lablup/backend.ai-fasttrack#3777 / PR lablup/backend.ai-fasttrack#3779

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.