lablup / lablup/backend.ai

Improve image pulling process

Open
#2,276 1 comment 0 reactions 1 assignee Claimed by @fregataa View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

# Steps

> [!NOTE]Step 1 targets 24.03, the remaining steps target 24.12.

## Step 1

- Allow superadmins to update status of kernels from `PULLING` to `CANCELLED` forcibly
## Step 2

- Implement `check-and-pull` APIIt checks whether an agent has a specific image or not.If the agent does not have the image, it starts a background task that pulls the image and returns the background task id.After the background task finishes, the agent should dispatch `ImagePulled` event.

- Check images early when creating kernelsWhen a kernel is scheduled, manager first calls the `check-and-pull` API. This agent-side API spawns a background task that verifies whether the agent has the required image. If the agent does not have the image, the task pulls it. The API produces an `ImagePullStartedEvent` when the agent begins pulling the image and an `ImagePullFinishedEvent` when the task completes.

- Consume Image eventsManager consumes `ImagePullStartedEvent` and `ImagePullFinishedEvent` events.When `ImagePullStartedEvent` consumed, transits kernel status to `PULLING`.When `ImagePullFinishedEvent` consumed, transits kernel status to `PREPARED`.

- `start-containers` loopManager runs a global asynchronous background loop that monitors all kernels with `PREPARED` status. It starts these kernels.

### Subtasks

- [x] #3128
- [ ] Update `USER_RESOURCE_OCCUPYING_KERNEL_STATUSES`Exclude `PULLING` kernel status from `USER_RESOURCE_OCCUPYING_KERNEL_STATUSES` since `PULLING` status kernels don't occupy any resources!
### As-is

```mermaid
sequenceDiagram
loop "prepare" loop
activate Manager
Manager->>Manager: Fetch "SCHEDULED" sessions
Manager->>+Agent: Create kernel
opt need to pull
Agent->>Event bus: Produce "Pulling" event
Agent->>Agent: Pull the image and wait till it finishes
Agent->>Event bus: Produce "Preparing" event
end
Agent->>Agent: Create kernel
Agent-->>-Manager: Return kernel creation info
deactivate Manager
end
```

### To-do

```mermaid
sequenceDiagram
loop "check-precond" loop
activate Manager
Manager->>Manager: Fetch "SCHEDULED" sessions and transit status to "PREPARING"
Manager->>Agent: check-and-pull
activate Agent
Note right of Agent: Run in background task
Agent-->>Manager: Return background task id
deactivate Manager
end
opt need to pull
Agent->>Event bus: Produce "Pulling" event
Agent->>Agent: Pull the image and wait till it finishes
end
Agent->>Event bus: Produce "Pull finished" event
deactivate Agent

loop "create-kernel" loop
activate Manager
Manager->>Manager: Fetch "PREPARED" sessions
Manager->>+Agent: Create kernel
Agent->>Agent: Create kernel
Agent-->>-Manager: Return kernel creation info
deactivate Manager
end
```

## Step 3

- Show image pulling progress (refs https://github.com/lablup/backend.ai/issues/227)
## Step 4

- Implement Image preload/unload API (refs https://github.com/lablup/backend.ai/issues/39)
- Create agents-images mapping table to let multiple agents preload images

JIRA Issue: BA-118

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.