lablup / lablup/backend.ai

Re-raise aiohttp connection errors as Backend.AI exceptions in client modules

Open
#6,306 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Problem

Currently, when manager attempts to communicate with agent or storage proxy servers, connection-related errors from aiohttp (such as `ClientConnectorError`) are not properly caught and converted to Backend.AI custom exceptions. This leads to inconsistent error handling and makes it difficult for upper layers to handle these errors uniformly.

## Current Situation

### Storage Proxy Client

- **Location**: `src/ai/backend/manager/clients/storage_proxy/base.py`
- **Issue**: `StorageProxyHTTPClient.request_stream_response()` directly calls `aiohttp.ClientSession.request()` without catching connection errors
- **Current error handling**: Only handles HTTP response status codes via `_handle_exceptional_response()`

### Agent RPC Client

- **Location**: `src/ai/backend/manager/clients/agent/`
- **Protocol**: Uses ZeroMQ RPC (not HTTP)
- **Current error handling**: `agent_cache.py`'s `rpc_context()` converts `AuthenticationError` to `RPCError`, but may need improvement for other connection-related errors

### App Proxy Client

- **Location**: `src/ai/backend/manager/clients/appproxy/`
- Similar HTTP-based client that may need the same treatment

## Existing Patterns

Some code already handles `ClientConnectorError`:

- `src/ai/backend/manager/api/scaling_group.py`: Only logs wsproxy connection failures
- `src/ai/backend/manager/services/artifact/service.py`: Retries on Reservoir connection failures and raises `ReservoirConnectionError`

However, these are inconsistent and not integrated into the client layer.

## Proposed Solution

1. **Storage Proxy Client**: Wrap `aiohttp.ClientSession.request()` calls with try-except to catch connection errors and re-raise as Backend.AI exceptions
1. **Agent Client**: Review and enhance connection error handling if needed
1. **App Proxy Client**: Apply similar error handling pattern
1. **Define new error types** if needed in `src/ai/backend/manager/errors/` for connection-related failures

## Acceptance Criteria

- [ ] Connection errors from storage proxy requests are caught and converted to appropriate Backend.AI exceptions
- [ ] Connection errors from agent requests are properly handled
- [ ] Connection errors from app proxy requests are properly handled
- [ ] Error messages include sufficient context (target server, operation being performed, etc.)
- [ ] Retry policies in resilience layer properly recognize these as retryable errors

JIRA Issue: BA-2745

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.