Population and removal of images in agents
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 14h 15m
- Merged PRs (30d)
- 335
Description
Let's add an admin-only ability to pre-populate and remove specific Docker images in agents.This moves the responsibility of managing availability of kernel images from the Docker registry to the manager. The Docker registry becomes a mere "repository" of images and the manager becomes the "gatekeeper" for image selection.
### Phase 1: A minimal background task framework
This is a simplified, light-weight background task framework, _without_ storing persistent task information in databases, but with live event broadcasts.
- [x] manager: New `manager.background` module\* Design and implement a generic background task interface\* `start_background_task(coro) -> bg_task_id`
- The coro should follow [a callback protocol](https://mypy.readthedocs.io/en/latest/protocols.html#callback-protocols)[1](1) to accept a progress reporter object, where it can ping "current progress" as a current value and a maximum value.
- Optionally let the callers use an aiojobs scheduler instance.\* If used, use its `close()` method when shutting down the manager.
- If not used, keep the reference of the asyncio task and cancel it upon shutting down.
- Use the event bus in `gateway.events` module to publish the progress reports as events.
- [x] manager: New API `/background-tasks?task_id={task_id`} in a `gateway.stream` module\* Accepts a single argument `task_id` as a query parameter.
- Subscribes the event bus for the given background task ID and sends them using the SSE response. (refer the existing `stream_events()` API handler)
[1](1): Callback protocols may be defined using either [typing.Callable](https://docs.python.org/3/library/typing.html#typing.Callable) or [typing.Protocol](https://docs.python.org/3/library/typing.html#typing.Protocol) with `__call__` method.
### Phase 2: Feature implementation
- [ ] manager: New GraphQL mutations `PreloadImage`, `UnloadImage`\* Accept two arguments: target agent set, target image reference.\* The agent set is a list of `"*"`, `"sgroup:{scaling_group_id}"`, `"{agent_id}"`.
- Only runnable by super-admins.
- Execute a background asyncio task in the request handler, with a random-generated task ID.
- Return the task ID as the API response so that clients can keep track of the progress.
- For removal, the manager should confirm if the image can be removed (i.e., not being used by any
- [ ] agent: New RPC handler to populate and remove images\* This should be implemented in the agent instead of the watcher because these operations are backend-dependent.
- It should get the required credentials to access the docker registry like auto-pulling mechanism in the kernel creation RPC function.
- First implement on the docker backend, using the docker API.
- Return an error if the docker daemon returns an error when there are containers using the target image.
```Java
### Phase 2
- [ ] #805
```
### Phase 3: Safety guards
- [ ] manager: New scheduler predicate in `manager.scheduler.predicates` module\* During the API operations, the manager's scheduler should block spawning of new sessions using that image. Add a new predicate that checks the image involved. Mark the kernel's `status_info` as `"pending-image-preparation"`.
- After finishing the operation, call the scheduler dispatcher to try rescheduling.
- [ ] manager: During download operations, `rescan-images` function must be blocked.\* We need to prevent populating different version of the target image due to rescanning in the middle of operation.
- This may require extension to the background task framework, such as categorization of background tasks using caller-provided key and checks for outstanding tasks. If so, let's store the list of outstanding tasks in a specific category as a simple set in Redis.\* Don't forget to remove the task ID from the category set in Redis when it's cancelled or finished.
### Phase 4: UI update
### Phase 5: docs update
### More advanced topics (postponed to implement later)
- What if the manager is going to be restarted?\* How long should we wait for?
- How to store the progress and resume the operation after restarts? (e.g., Celery, Redis Streams, etc.)
- Should be the download/removal operations cancellable?\* It is not practical to revert partially done operations. Let's just warn the admins in the manual and let the operations stop.
- If an agent is pulling the image and the download operation is cancelled, the agent's result report may be accumulated in some queue brokers (e.g., Celery with RabbitMQ) and cause memory leaks as no one reclaims it. This should be actively avoided (e.g., by setting expiration timeout).
### Related issues
- lablup/backend.ai-manager#84
- lablup/backend.ai-manager#110
┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1159751085623729/1159757172069100) by [Unito](https://www.unito.io/learn-more)
JIRA Issue: BA-324
Contributor guide
Assessment
This issue has not been assessed yet.