lablup / lablup/backend.ai

Improve model service autoscaling

Open
#3,496 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Motivation  

Let’s follow-up some incomplete corners of [https://lablup.atlassian.net/browse/BA-96](https://lablup.atlassian.net/browse/BA-96)

## Expected Sub Issue

1. Define clearer priority semantics when there are multiple rules to be triggered at the same time. Currently only the first matched rule is evaluated, but if there are multiple rules observing different metrics, they need to be evaluated in a single iteration and _somehow_ the results must be combined.
1. We could consider having a more sincerely designed validation of autoscaling rules for a single endpoint. For instance, only a single pair of increasing/decreasing rules may exist against a single metric. If so, we could group the rules by metrics and evaluate each group simultaneously, and prioritize their results using a configured order.
1. Support additional aggregation operators when collecting metrics from multiple replica sessions and kernels, as currently we have only “average”. (e.g., min, max)
1. Like idle checkers, we need to consider having time-based, windowed metric smoothing.
1. Users would want to have a GUI to see the current metrics.
1. Leave user-queryable explicit audit logging of the scaling decisions.
1. Consider adding the endpoint-level cool-down, in addition to individual rules.
1. Allow disabling a specific autoscaling rule without deleting it.

JIRA Issue: BA-537

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.