vllm-project / vllm-project/aibrix
Support predictive autoscaling for LLM inference
Open
area/autoscaling
kind/feature
priority/important-soon
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 697
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 104
Description
### 🚀 Feature Description and Motivation
We used to do a lot of work around reactive autoscaling, however, the model bootstrap still take looks long time. Instead of traditional autoscaling, we want to provide another option - the time series prediction method to solve the latency issue.
### Use Case
LLM autoscaling
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.