Improve model service autoscaling
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Motivation
Let’s follow-up some incomplete corners of [https://lablup.atlassian.net/browse/BA-96](https://lablup.atlassian.net/browse/BA-96)
## Expected Sub Issue
1. Define clearer priority semantics when there are multiple rules to be triggered at the same time. Currently only the first matched rule is evaluated, but if there are multiple rules observing different metrics, they need to be evaluated in a single iteration and _somehow_ the results must be combined.
1. We could consider having a more sincerely designed validation of autoscaling rules for a single endpoint. For instance, only a single pair of increasing/decreasing rules may exist against a single metric. If so, we could group the rules by metrics and evaluate each group simultaneously, and prioritize their results using a configured order.
1. Support additional aggregation operators when collecting metrics from multiple replica sessions and kernels, as currently we have only “average”. (e.g., min, max)
1. Like idle checkers, we need to consider having time-based, windowed metric smoothing.
1. Users would want to have a GUI to see the current metrics.
1. Leave user-queryable explicit audit logging of the scaling decisions.
1. Consider adding the endpoint-level cool-down, in addition to individual rules.
1. Allow disabling a specific autoscaling rule without deleting it.
JIRA Issue: BA-537
Contributor guide
Assessment
This issue has not been assessed yet.