lablup / lablup/backend.ai

Refactor session preparation as a queue-based operation

Open
#515 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Currently the session creation steps is composed of two async tasks:

- `PENDING` → `SCHEDULED`: Per each scaling group, we determine which sessions to start using the scheduler logic and a global lock. The scheduling decision is made serially by updating the remaining resources of target agents inside a big transaction.
- `SCHEDULED` → `PREPARING`: Per each scaling group, we make agent RPC calls and process the results. When there are multiple sessions to start upon invocation, it uses `asyncio.PersistentTaskGroup()` to run the operations concurrently.
Historically these two steps were executed in a single coroutine task, but we later split them (lablup/backend.ai-manager#415) to avoid excessively lengthy transactions due to RPC delays in the agents. (The first scheduling-decision step is relatively fast because it is a purely logical operation inside the manager and database.)

Now, we have some cases that reveals the above medication was not enough.

When there is a failure in a specific session start task (this is fixed in #514) or there is a global-lock (`LOCKID_PREPARE`) expiration timeout, all other session start tasks for the given scaling group are interrupted together. This is what we don't want, because start failures should be isolated session by session (and thus tenant by tenant).

As provisioning of sessions may take time, they should be not only asynchronously executed but also have isolation of failures and explicit queuing to monitor the system health status with ease. For example, we could make a metric just like [Travis CI's build backlog](https://www.traviscistatus.com/) from the length of this "PREPARE" queue (Note that this metric will be further enhanced with #412!):

So, let's make the session start tasks to run via a central queue based on the event bus subsystem.

JIRA Issue: BA-262

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.