lablup / lablup/backend.ai

Fix sparse file creation failure: add cleanup on error and use proper BackendAI exception

Open
#10,327 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Problem

Kernel creation fails with `RuntimeError: could not create sparse file` in `_create_loop_filesystem` when `scratch_type = "hostfile"`. There are three distinct issues:

### 1. Pre-existing file not handled (`_create_sparse_file`)

`touch(exist_ok=True)` silently opens an existing file, and `os.truncate` does NOT deallocate already-allocated blocks. If a previous failed attempt left a scratch file with actual blocks (e.g., after `mkfs.ext4` ran), the `st_blocks != 0` check raises an error on retry.

\***Fix**\*: Check if the file exists before creation; if it does, unlink and recreate instead of reusing.

### 2. No cleanup on failure (`_create_loop_filesystem`)

When `mkfs.ext4` or `mount` fails, the scratch file and scratch directory are left on disk. The outer `except Exception` handler in `create_kernel` only calls `reconstruct_resource_usage()` — it does not clean up scratch artifacts.

\***Fix**\*: Add a try/except in `_create_loop_filesystem` that unlinks `scratch_file` and removes `scratch_dir` on any failure.

### 3. `RuntimeError` violates agent coding rules

`agent/CLAUDE.md` requires all business logic exceptions to inherit from `BackendAIError`. `RuntimeError` is raised directly in three places inside `scratch.py`.

\***Fix**\*: Define `ScratchFileCreationError`, `MkfsError`, and `ScratchMountError` (or a single `ScratchSetupError`) in `agent/errors/kernel.py`, following the existing pattern.

## Related

- Relates to BA-1409 (Refactor Agent's Kernel Creation Flow — cleanup on resource setup failure)
- See also BA-2909 (sparse file failure for specific images)

## Target Files

- `src/ai/backend/agent/stage/kernel_lifecycle/docker/scratch.py`
- `src/ai/backend/agent/errors/kernel.py`

## Success Criteria

- [ ] With `hostfile` scratch type, kernel creation succeeds even when a leftover scratch file with allocated blocks exists from a previous failed attempt
- [ ] When `mkfs.ext4` or `mount` fails inside `_create_loop_filesystem`, both `scratch_file` and `scratch_dir` are removed automatically
- [ ] `RuntimeError` is replaced with `BackendAIError` subclass(es) defined in `agent/errors/kernel.py`
- [ ] `pants lint`, `pants check`, and `pants test` pass for the affected packages

JIRA Issue: BA-5300

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.