Fix sparse file creation failure: add cleanup on error and use proper BackendAI exception
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Problem
Kernel creation fails with `RuntimeError: could not create sparse file` in `_create_loop_filesystem` when `scratch_type = "hostfile"`. There are three distinct issues:
### 1. Pre-existing file not handled (`_create_sparse_file`)
`touch(exist_ok=True)` silently opens an existing file, and `os.truncate` does NOT deallocate already-allocated blocks. If a previous failed attempt left a scratch file with actual blocks (e.g., after `mkfs.ext4` ran), the `st_blocks != 0` check raises an error on retry.
\***Fix**\*: Check if the file exists before creation; if it does, unlink and recreate instead of reusing.
### 2. No cleanup on failure (`_create_loop_filesystem`)
When `mkfs.ext4` or `mount` fails, the scratch file and scratch directory are left on disk. The outer `except Exception` handler in `create_kernel` only calls `reconstruct_resource_usage()` — it does not clean up scratch artifacts.
\***Fix**\*: Add a try/except in `_create_loop_filesystem` that unlinks `scratch_file` and removes `scratch_dir` on any failure.
### 3. `RuntimeError` violates agent coding rules
`agent/CLAUDE.md` requires all business logic exceptions to inherit from `BackendAIError`. `RuntimeError` is raised directly in three places inside `scratch.py`.
\***Fix**\*: Define `ScratchFileCreationError`, `MkfsError`, and `ScratchMountError` (or a single `ScratchSetupError`) in `agent/errors/kernel.py`, following the existing pattern.
## Related
- Relates to BA-1409 (Refactor Agent's Kernel Creation Flow — cleanup on resource setup failure)
- See also BA-2909 (sparse file failure for specific images)
## Target Files
- `src/ai/backend/agent/stage/kernel_lifecycle/docker/scratch.py`
- `src/ai/backend/agent/errors/kernel.py`
## Success Criteria
- [ ] With `hostfile` scratch type, kernel creation succeeds even when a leftover scratch file with allocated blocks exists from a previous failed attempt
- [ ] When `mkfs.ext4` or `mount` fails inside `_create_loop_filesystem`, both `scratch_file` and `scratch_dir` are removed automatically
- [ ] `RuntimeError` is replaced with `BackendAIError` subclass(es) defined in `agent/errors/kernel.py`
- [ ] `pants lint`, `pants check`, and `pants test` pass for the affected packages
JIRA Issue: BA-5300
Contributor guide
Assessment
This issue has not been assessed yet.