lablup / lablup/backend.ai

Show the load-balancing dispersion ratio in model services

Open
#3,705 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Motivation  

Each model service endpoint may be backed by multiple replica sessions (either auto-scaled or manually-scaled). Depending on the scheduling decision, they may be concentrated in a small number of agents or scattered around a large number of agents.

The users may want to know if their model services are “sufficiently” distributed to avoid SPoF. Since we are hiding the physical node details by default, there is no way to check how they are distributed from the user’s perspective without the admin privilege.

Let’s provide a summarization metric to show this dispersion level to the user.

## Required Features

The “dispersion level” is calculated as:

```
(dispersion level) = (the union number of agents used by the endpoint) / (the number of replica sessions)
```

**Example:**

- All replica sessions are running non-overlapping, separate agents: dispersion-level = 1.
- Two of three replica sessions are running on a same agent while the other on another agent: dispersion-level = 2/3 = 0.667

## Additional Consideration

### Cluster Sessions

When each replica is a cluster session having multiple kernel containers, there may be multiple different definitions of the dispersion level:

- `(the union number of agents used by all kernels of all replica sessions) / (the number of replica sessions)`
- Example: 3 clustered replica sessions with 4 kernels for each, using 2 agents in total → 0.667
- `(the union number of agents used by all kernels of all replica sessions) / (the number of all kernels of all replica sessions)`
- Example: 3 clustered replica sessions with 4 kernels for each, using 2 agents in total → 2 / (4 x 3) = 0.167
- Example: 3 clustered replica sessions with 4 kernels for each, using 2 agents by each replica disjointly → (2 x 3) / (4 x 3) = 0.5
- `average of (the union number of agents used by all kernels of each replica session) / (the number of all kernels of each replica session)`
- Example: 3 clustered replica sessions with 4 kernels for each, using 2 agents in the 1st replica, 3 agents in the 2nd replica, 4 agents in the 3rd replica → avg(0.5, 0.75, 1) = 0.75

We need to clarify this before writing codes.

### New API or GraphQL schema?

With `hide-agents = true`, the user cannot query the exact agent running their sessions. Therefore, we need a dedicated GraphQL field or API to retrieve this number as a “summarization” of the underlying hardware information.

JIRA Issue: BA-755

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.