lablup / lablup/backend.ai

Agent: Implement mountpoint health checker for monitoring storage mount availability

Open
#7,106 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Overview

Implement a health checker component that monitors the availability and accessibility of storage mountpoints used for vfolder mounts.

## Parent Epic

BA-3244

## Background

The Agent mounts vfolders to containers at runtime. These mounts depend on storage being properly mounted at the agent level (configured via `mount_path` in agent config). When the underlying storage becomes unavailable (e.g., NFS server down, network issues), kernel creation will fail.

## Requirements

### Functional Requirements

1. **Mountpoint Status Checking**
- Check if configured mount paths exist and are accessible
- Verify read/write permissions on mount directories
- Detect stale NFS mounts or hung filesystems
1. **Health Status Reporting**
- Maintain current mountpoint health status
- Provide API to query mountpoint status
- Support multiple mountpoints if configured
1. **Periodic Monitoring**
- Configurable check interval
- Handle check timeouts (stale mounts can cause hangs)

### Non-Functional Requirements

- Minimal performance impact during health checks
- Timeout handling for hung filesystem operations
- Thread-safe status access

## Technical Design

### Location

- `src/ai/backend/agent/health/mountpoint.py` (new file)

### Key Components

```python
@dataclass(frozen=True)
class MountpointStatus:
path: Path
available: bool
readable: bool
writable: bool
last_checked: datetime
error_message: Optional[str] = None

class MountpointHealthChecker:
async def check_mountpoint(self, path: Path) -> MountpointStatus: ...
async def get_all_statuses(self) -> list[MountpointStatus]: ...
async def is_all_healthy(self) -> bool: ...
```

### Integration Points

- Agent startup: Initialize health checker with configured mount paths
- Health check endpoint: Expose mountpoint status via agent health API
- Pre-creation checks: Query status before kernel creation

## Acceptance Criteria

- [ ] Health checker detects when a mountpoint is unavailable
- [ ] Health checker detects when a mountpoint is read-only
- [ ] Health checker handles stale/hung mount operations with timeout
- [ ] Status is queryable via API
- [ ] Periodic checks run in background without blocking agent operations

## Related Files

- `src/ai/backend/agent/config/unified.py`: Agent configuration including mount_path
- `src/ai/backend/agent/health/docker.py`: Existing health checker pattern
- `src/ai/backend/agent/server.py`: Agent server initialization

JIRA Issue: BA-3245

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.