lablup / lablup/backend.ai

Agent: Add non-retryable pre-creation condition check for mountpoint availability

Open
#7,107 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Overview

Add pre-creation condition validation in the kernel creation flow that checks mountpoint availability and fails with a non-retryable error when storage mounts are unavailable.

## Parent Epic

BA-3244

## Background

Currently, kernel creation proceeds even when storage mounts are unavailable, leading to failures during the vfolder mounting phase. These failures should be detected earlier and reported as non-retryable errors so that:

1. The failure is immediate and informative
1. The scheduler doesn't retry on the same agent
1. Operators can quickly identify storage issues

## Requirements

### Functional Requirements

1. **Pre-creation Validation**
- Check mountpoint health status before proceeding with kernel creation
- Validate all required mount paths for the kernel's vfolders
1. **Non-retryable Error Response**
- Return error with clear indication that retry won't help
- Include specific mountpoint that failed
- Include diagnostic information (path, error type)
1. **Error Information Propagation**
- Error should be captured by Manager
- Error should be visible in session history
- Error should enable appropriate scheduling decisions

### Error Types

- `MountpointUnavailableError`: Base error for mount issues
- `MountpointNotFoundError`: Mount path doesn't exist
- `MountpointNotAccessibleError`: Mount path exists but is inaccessible
- `MountpointReadOnlyError`: Mount path is read-only when write is needed

## Technical Design

### Location

- `src/ai/backend/agent/exception.py`: Add new exception types
- `src/ai/backend/agent/agent.py`: Add pre-creation check in `create_kernel` flow

### Integration Point

In `create_kernel` flow (around line 2782-2790 in agent.py):

```python
# Before mounting vfolders
vfolder_mounts = [VFolderMount.from_json(item) for item in kernel_config["mounts"]]

# NEW: Pre-creation check
await self._validate_mountpoint_availability(vfolder_mounts)

if not restarting:
await ctx.mount_vfolders(vfolder_mounts, resource_spec)
```

### Error Response Format

```python
@dataclass
class KernelCreationPreConditionError:
error_type: str # "mountpoint_unavailable", etc.
retryable: bool # Always False for mountpoint issues
message: str
details: dict[str, Any] # path, reason, etc.
```

## Acceptance Criteria

- [ ] Kernel creation fails fast when mountpoint is unavailable
- [ ] Error response clearly indicates non-retryable status
- [ ] Error includes specific mountpoint path and failure reason
- [ ] Error is properly propagated to Manager
- [ ] Session history shows appropriate error message

## Dependencies

- Mountpoint health checker implementation (Story 1)

## Related Files

- `src/ai/backend/agent/agent.py`: Kernel creation flow
- `src/ai/backend/agent/exception.py`: Agent exceptions
- `src/ai/backend/common/types.py`: Common error types

JIRA Issue: BA-3246

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.