Refactor session preparation as a queue-based operation
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
Currently the session creation steps is composed of two async tasks:
- `PENDING` → `SCHEDULED`: Per each scaling group, we determine which sessions to start using the scheduler logic and a global lock. The scheduling decision is made serially by updating the remaining resources of target agents inside a big transaction.
- `SCHEDULED` → `PREPARING`: Per each scaling group, we make agent RPC calls and process the results. When there are multiple sessions to start upon invocation, it uses `asyncio.PersistentTaskGroup()` to run the operations concurrently.
Historically these two steps were executed in a single coroutine task, but we later split them (lablup/backend.ai-manager#415) to avoid excessively lengthy transactions due to RPC delays in the agents. (The first scheduling-decision step is relatively fast because it is a purely logical operation inside the manager and database.)
Now, we have some cases that reveals the above medication was not enough.
When there is a failure in a specific session start task (this is fixed in #514) or there is a global-lock (`LOCKID_PREPARE`) expiration timeout, all other session start tasks for the given scaling group are interrupted together. This is what we don't want, because start failures should be isolated session by session (and thus tenant by tenant).
As provisioning of sessions may take time, they should be not only asynchronously executed but also have isolation of failures and explicit queuing to monitor the system health status with ease. For example, we could make a metric just like [Travis CI's build backlog](https://www.traviscistatus.com/) from the length of this "PREPARE" queue (Note that this metric will be further enhanced with #412!):
So, let's make the session start tasks to run via a central queue based on the event bus subsystem.
JIRA Issue: BA-262
Contributor guide
Assessment
This issue has not been assessed yet.