Show the load-balancing dispersion ratio in model services
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Motivation
Each model service endpoint may be backed by multiple replica sessions (either auto-scaled or manually-scaled). Depending on the scheduling decision, they may be concentrated in a small number of agents or scattered around a large number of agents.
The users may want to know if their model services are “sufficiently” distributed to avoid SPoF. Since we are hiding the physical node details by default, there is no way to check how they are distributed from the user’s perspective without the admin privilege.
Let’s provide a summarization metric to show this dispersion level to the user.
## Required Features
The “dispersion level” is calculated as:
```
(dispersion level) = (the union number of agents used by the endpoint) / (the number of replica sessions)
```
**Example:**
- All replica sessions are running non-overlapping, separate agents: dispersion-level = 1.
- Two of three replica sessions are running on a same agent while the other on another agent: dispersion-level = 2/3 = 0.667
## Additional Consideration
### Cluster Sessions
When each replica is a cluster session having multiple kernel containers, there may be multiple different definitions of the dispersion level:
- `(the union number of agents used by all kernels of all replica sessions) / (the number of replica sessions)`
- Example: 3 clustered replica sessions with 4 kernels for each, using 2 agents in total → 0.667
- `(the union number of agents used by all kernels of all replica sessions) / (the number of all kernels of all replica sessions)`
- Example: 3 clustered replica sessions with 4 kernels for each, using 2 agents in total → 2 / (4 x 3) = 0.167
- Example: 3 clustered replica sessions with 4 kernels for each, using 2 agents by each replica disjointly → (2 x 3) / (4 x 3) = 0.5
- `average of (the union number of agents used by all kernels of each replica session) / (the number of all kernels of each replica session)`
- Example: 3 clustered replica sessions with 4 kernels for each, using 2 agents in the 1st replica, 3 agents in the 2nd replica, 4 agents in the 3rd replica → avg(0.5, 0.75, 1) = 0.75
We need to clarify this before writing codes.
### New API or GraphQL schema?
With `hide-agents = true`, the user cannot query the exact agent running their sessions. Therefore, we need a dedicated GraphQL field or API to retrieve this number as a “summarization” of the underlying hardware information.
JIRA Issue: BA-755
Contributor guide
Assessment
This issue has not been assessed yet.