lablup / lablup/backend.ai

Propagate Detailed Error Messages from Storage Proxy and Agent to Client

Open
#7,698 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

# Motivation

Currently, when exceptions occur in the Storage Proxy or Agent components, the Manager only displays generic and unhelpful error messages such as "Unexpected error" to users. This makes it extremely difficult for users and administrators to diagnose and troubleshoot issues, as they have no visibility into the actual root cause of failures.

For example, when a file upload fails due to storage quota exceeded, or when a kernel launch fails due to resource constraints on the agent, users only see a vague error message without any actionable information. This leads to:

- Increased support burden as users cannot self-diagnose issues
- Longer time-to-resolution for operational problems
- Poor user experience and frustration
- Difficulty in debugging production issues

Propagating meaningful error details from downstream services (Storage Proxy, Agent) through the Manager to the Client will significantly improve observability and user experience.

----

# Required Features

1. **Error Classification and Propagation Framework**
- Define a standardized error response format that can carry detailed error information across service boundaries
- Include error code, error type, human-readable message, and optional context/metadata
1. **Storage Proxy Error Propagation**
- Update Storage Proxy HTTP client to capture and forward detailed error responses
- Preserve original error messages, codes, and context when relaying errors to Manager
- Handle storage-specific errors (quota exceeded, permission denied, file not found, etc.)
1. **Agent Error Propagation**
- Update Agent RPC communication to include detailed error information
- Propagate kernel-related errors (resource exhaustion, image pull failures, execution errors, etc.)
- Maintain error chain/stack for debugging purposes
1. **Manager Error Handling Enhancement**
- Update Manager's error handling to extract and forward detailed error information from downstream services
- Avoid masking original errors with generic "Unexpected error" messages
- Implement proper error translation layer that preserves meaningful details while sanitizing sensitive information
1. **Client-Facing Error Response**
- Ensure API responses include detailed, actionable error messages
- Maintain backward compatibility with existing error response format
- Add optional verbose mode for debugging scenarios

----

# **Affected Areas:**

- `src/ai/backend/storage/` - Storage Proxy error handling
- `src/ai/backend/agent/` - Agent error responses
- `src/ai/backend/manager/api/` - Manager API error handling
- `src/ai/backend/manager/clients/` - Service client error handling
- `src/ai/backend/common/` - Shared error types and utilities
- `src/ai/backend/client/` - Client SDK error parsing

----

# Testing Scenarios

## Storage Proxy Error Propagation

- [ ] Trigger storage quota exceeded error and verify detailed message reaches client
- [ ] Trigger permission denied error and verify proper error code and message
- [ ] Trigger file not found error and verify path information is included
- [ ] Verify sensitive information (internal paths, credentials) is sanitized

## Agent Error Propagation

- [ ] Trigger kernel launch failure due to insufficient resources and verify detailed error
- [ ] Trigger image pull failure and verify error message includes image name and reason
- [ ] Trigger execution timeout and verify timeout details are propagated
- [ ] Verify agent internal errors don't leak sensitive system information

## Manager Error Handling

- [ ] Verify Manager correctly forwards Storage Proxy errors to client
- [ ] Verify Manager correctly forwards Agent errors to client
- [ ] Verify error chain is preserved for debugging
- [ ] Verify backward compatibility with existing error response format

## End-to-End Scenarios

- [ ] File upload failure shows meaningful error to user
- [ ] Session creation failure shows specific reason (resource, image, config issue)
- [ ] Verify error messages are user-friendly and actionable
- [ ] Verify logs contain full error details for administrator debugging

## Security Testing

- [ ] Verify internal stack traces are not exposed to clients
- [ ] Verify internal file paths are sanitized
- [ ] Verify no credential or token leakage in error messages

JIRA Issue: BA-3642

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.