Agent: Implement mountpoint health checker for monitoring storage mount availability
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Overview
Implement a health checker component that monitors the availability and accessibility of storage mountpoints used for vfolder mounts.
## Parent Epic
BA-3244
## Background
The Agent mounts vfolders to containers at runtime. These mounts depend on storage being properly mounted at the agent level (configured via `mount_path` in agent config). When the underlying storage becomes unavailable (e.g., NFS server down, network issues), kernel creation will fail.
## Requirements
### Functional Requirements
1. **Mountpoint Status Checking**
- Check if configured mount paths exist and are accessible
- Verify read/write permissions on mount directories
- Detect stale NFS mounts or hung filesystems
1. **Health Status Reporting**
- Maintain current mountpoint health status
- Provide API to query mountpoint status
- Support multiple mountpoints if configured
1. **Periodic Monitoring**
- Configurable check interval
- Handle check timeouts (stale mounts can cause hangs)
### Non-Functional Requirements
- Minimal performance impact during health checks
- Timeout handling for hung filesystem operations
- Thread-safe status access
## Technical Design
### Location
- `src/ai/backend/agent/health/mountpoint.py` (new file)
### Key Components
```python
@dataclass(frozen=True)
class MountpointStatus:
path: Path
available: bool
readable: bool
writable: bool
last_checked: datetime
error_message: Optional[str] = None
class MountpointHealthChecker:
async def check_mountpoint(self, path: Path) -> MountpointStatus: ...
async def get_all_statuses(self) -> list[MountpointStatus]: ...
async def is_all_healthy(self) -> bool: ...
```
### Integration Points
- Agent startup: Initialize health checker with configured mount paths
- Health check endpoint: Expose mountpoint status via agent health API
- Pre-creation checks: Query status before kernel creation
## Acceptance Criteria
- [ ] Health checker detects when a mountpoint is unavailable
- [ ] Health checker detects when a mountpoint is read-only
- [ ] Health checker handles stale/hung mount operations with timeout
- [ ] Status is queryable via API
- [ ] Periodic checks run in background without blocking agent operations
## Related Files
- `src/ai/backend/agent/config/unified.py`: Agent configuration including mount_path
- `src/ai/backend/agent/health/docker.py`: Existing health checker pattern
- `src/ai/backend/agent/server.py`: Agent server initialization
JIRA Issue: BA-3245
Contributor guide
Assessment
This issue has not been assessed yet.