vllm-project / vllm-project/aibrix

Support predictive autoscaling for LLM inference

Open
#1,418 8 comments 0 reactions 1 assignee Claimed by @Belyenochi View on GitHub
area/autoscaling kind/feature priority/important-soon
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

### 🚀 Feature Description and Motivation

We used to do a lot of work around reactive autoscaling, however, the model bootstrap still take looks long time. Instead of traditional autoscaling, we want to provide another option - the time series prediction method to solve the latency issue.

### Use Case

LLM autoscaling

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.