Re-raise aiohttp connection errors as Backend.AI exceptions in client modules
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Problem
Currently, when manager attempts to communicate with agent or storage proxy servers, connection-related errors from aiohttp (such as `ClientConnectorError`) are not properly caught and converted to Backend.AI custom exceptions. This leads to inconsistent error handling and makes it difficult for upper layers to handle these errors uniformly.
## Current Situation
### Storage Proxy Client
- **Location**: `src/ai/backend/manager/clients/storage_proxy/base.py`
- **Issue**: `StorageProxyHTTPClient.request_stream_response()` directly calls `aiohttp.ClientSession.request()` without catching connection errors
- **Current error handling**: Only handles HTTP response status codes via `_handle_exceptional_response()`
### Agent RPC Client
- **Location**: `src/ai/backend/manager/clients/agent/`
- **Protocol**: Uses ZeroMQ RPC (not HTTP)
- **Current error handling**: `agent_cache.py`'s `rpc_context()` converts `AuthenticationError` to `RPCError`, but may need improvement for other connection-related errors
### App Proxy Client
- **Location**: `src/ai/backend/manager/clients/appproxy/`
- Similar HTTP-based client that may need the same treatment
## Existing Patterns
Some code already handles `ClientConnectorError`:
- `src/ai/backend/manager/api/scaling_group.py`: Only logs wsproxy connection failures
- `src/ai/backend/manager/services/artifact/service.py`: Retries on Reservoir connection failures and raises `ReservoirConnectionError`
However, these are inconsistent and not integrated into the client layer.
## Proposed Solution
1. **Storage Proxy Client**: Wrap `aiohttp.ClientSession.request()` calls with try-except to catch connection errors and re-raise as Backend.AI exceptions
1. **Agent Client**: Review and enhance connection error handling if needed
1. **App Proxy Client**: Apply similar error handling pattern
1. **Define new error types** if needed in `src/ai/backend/manager/errors/` for connection-related failures
## Acceptance Criteria
- [ ] Connection errors from storage proxy requests are caught and converted to appropriate Backend.AI exceptions
- [ ] Connection errors from agent requests are properly handled
- [ ] Connection errors from app proxy requests are properly handled
- [ ] Error messages include sufficient context (target server, operation being performed, etc.)
- [ ] Retry policies in resilience layer properly recognize these as retryable errors
JIRA Issue: BA-2745
Contributor guide
Assessment
This issue has not been assessed yet.