alibaba / alibaba/ROCK

[Bug] fix(scheduler): ImageCleanupTask silently skips dangling/BuildKit prune after first run

Open
#995 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
485
Forks
81
Avg merge
16h 12m
Merged PRs (30d)
8

Description

## Summary

After merged dangling-layer + BuildKit prune into
`ImageCleanupTask`, the prune steps are silently skipped on every trigger
**after the first** on a given worker. `run_report.json` still reports
`success_count: 1, failed_count: 0`. **Image / BuildKit cache cleanup is
effectively disabled** on long-running workers, while operators see
"success" in the scheduler report.

## Reproduction

1. Pick a worker IP. First-trigger ImageCleanupTask:
```bash
/trigger image_cleanup
```
docuum daemon starts, dangling prune runs, dangling=0 on the worker.

2. Manually create a dangling image on the worker:
```bash
echo "FROM busybox" > /tmp/Dockerfile
echo "RUN echo v1 > /tmp/x" >> /tmp/Dockerfile
docker build -t test /tmp
sed -i 's/v1/v2/' /tmp/Dockerfile
docker build -t test /tmp
docker images --filter dangling=true -q | wc -l # 1
```

3. Trigger ImageCleanupTask again:
```bash
/trigger image_cleanup
cat /data/scheduler_status/image_cleanup_run_report.json
# success_count: 1, failed_count: 0
```

4. Re-check on worker:
```bash
docker images --filter dangling=true -q | wc -l # still 1, NOT cleaned
```

5. Manually run the same command from inside the task:
```bash
docker image prune -f --filter dangling=true
# Deleted Images: sha256:... ✅ works fine
```

The prune command is correct; it just never reaches the worker after the
first trigger.

## Root cause

`rock/admin/scheduler/task_base.py:216-234`, `should_run()`:

```python
async def should_run(self, runtime) -> bool:
if self.idempotency == IdempotencyType.IDEMPOTENT:
return True
# NON_IDEMPOTENT path
status = await self.get_task_status(runtime)
if status is None:
return True
if status.pid and status.status == TaskStatusEnum.RUNNING:
pid_exists = await runtime.check_pid_exists(status.pid)
if pid_exists:
return False # whole task skipped
return True
```

`ImageCleanupTask` declares `idempotency = NON_IDEMPOTENT` because it
launches a long-running docuum daemon. After the first trigger:
- worker's status file has `pid=, status=RUNNING`
- docuum is still running (it's a daemon, expected behavior)
- `check_pid_exists` returns True → `should_run` returns False
- `run_on_worker` short-circuits, `run_action` is never called
- the prune steps inside `run_action` (added by #970) are skipped along
with the docuum start

But `BaseTask.run` still records `(ip, True, None)` in `success_ips`, so
the run report assert `success_count: 1` — masking the silent failure.

## Why no test caught it

PR #970 added unit tests at
`tests/unit/admin/scheduler/test_image_cleanup_task.py`, but they call
`task.run_action(runtime)` directly, **bypassing the `should_run` gate**
upstream. End-to-end `task.run([ips])` goes through `run_on_worker →
should_run → single_run → run_action`; only that path is affected.

## Affected scope

| Scenario | Affected? |
|---|---|
| Dangling-image cleanup on long-running workers | **Yes** (silently) |
| BuildKit cache trim with `keep_build_storage` | **Yes** (silently) |
| docuum LRU image eviction | No (still runs, but never re-launches) |
| Single-trigger / first deploy | No |
| Workers that just restarted (status file gone) | No (resets to first-trigger) |

Severity in production: dangling layers and BuildKit cache accumulate
indefinitely. Until manual SRE intervention or worker pod restart,
cleanup is dead.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read rock/admin/scheduler/task_base.py:216-234 and the ImageCleanupTask idempotency declaration, then trace should_run through run_on_worker and run_action. Reproduce the issue with repeated /trigger image_cleanup calls and inspect image_cleanup_run_report.json. Done means later triggers reach the cleanup steps, remove dangling images, and report their actual result.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.