[Bug] fix(scheduler): ImageCleanupTask silently skips dangling/BuildKit prune after first run
- Dominant language
- Python
- Stars
- 485
- Forks
- 81
- Avg merge
- 16h 12m
- Merged PRs (30d)
- 8
Description
## Summary
After merged dangling-layer + BuildKit prune into
`ImageCleanupTask`, the prune steps are silently skipped on every trigger
**after the first** on a given worker. `run_report.json` still reports
`success_count: 1, failed_count: 0`. **Image / BuildKit cache cleanup is
effectively disabled** on long-running workers, while operators see
"success" in the scheduler report.
## Reproduction
1. Pick a worker IP. First-trigger ImageCleanupTask:
```bash
/trigger image_cleanup
```
docuum daemon starts, dangling prune runs, dangling=0 on the worker.
2. Manually create a dangling image on the worker:
```bash
echo "FROM busybox" > /tmp/Dockerfile
echo "RUN echo v1 > /tmp/x" >> /tmp/Dockerfile
docker build -t test /tmp
sed -i 's/v1/v2/' /tmp/Dockerfile
docker build -t test /tmp
docker images --filter dangling=true -q | wc -l # 1
```
3. Trigger ImageCleanupTask again:
```bash
/trigger image_cleanup
cat /data/scheduler_status/image_cleanup_run_report.json
# success_count: 1, failed_count: 0
```
4. Re-check on worker:
```bash
docker images --filter dangling=true -q | wc -l # still 1, NOT cleaned
```
5. Manually run the same command from inside the task:
```bash
docker image prune -f --filter dangling=true
# Deleted Images: sha256:... ✅ works fine
```
The prune command is correct; it just never reaches the worker after the
first trigger.
## Root cause
`rock/admin/scheduler/task_base.py:216-234`, `should_run()`:
```python
async def should_run(self, runtime) -> bool:
if self.idempotency == IdempotencyType.IDEMPOTENT:
return True
# NON_IDEMPOTENT path
status = await self.get_task_status(runtime)
if status is None:
return True
if status.pid and status.status == TaskStatusEnum.RUNNING:
pid_exists = await runtime.check_pid_exists(status.pid)
if pid_exists:
return False # whole task skipped
return True
```
`ImageCleanupTask` declares `idempotency = NON_IDEMPOTENT` because it
launches a long-running docuum daemon. After the first trigger:
- worker's status file has `pid=, status=RUNNING`
- docuum is still running (it's a daemon, expected behavior)
- `check_pid_exists` returns True → `should_run` returns False
- `run_on_worker` short-circuits, `run_action` is never called
- the prune steps inside `run_action` (added by #970) are skipped along
with the docuum start
But `BaseTask.run` still records `(ip, True, None)` in `success_ips`, so
the run report assert `success_count: 1` — masking the silent failure.
## Why no test caught it
PR #970 added unit tests at
`tests/unit/admin/scheduler/test_image_cleanup_task.py`, but they call
`task.run_action(runtime)` directly, **bypassing the `should_run` gate**
upstream. End-to-end `task.run([ips])` goes through `run_on_worker →
should_run → single_run → run_action`; only that path is affected.
## Affected scope
| Scenario | Affected? |
|---|---|
| Dangling-image cleanup on long-running workers | **Yes** (silently) |
| BuildKit cache trim with `keep_build_storage` | **Yes** (silently) |
| docuum LRU image eviction | No (still runs, but never re-launches) |
| Single-trigger / first deploy | No |
| Workers that just restarted (status file gone) | No (resets to first-trigger) |
Severity in production: dangling layers and BuildKit cache accumulate
indefinitely. Until manual SRE intervention or worker pod restart,
cleanup is dead.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read rock/admin/scheduler/task_base.py:216-234 and the ImageCleanupTask idempotency declaration, then trace should_run through run_on_worker and run_action. Reproduce the issue with repeated /trigger image_cleanup calls and inspect image_cleanup_run_report.json. Done means later triggers reach the cleanup steps, remove dangling images, and report their actual result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100