vllm-project / vllm-project/aibrix

Adding scaling down to 0 case Gateway handling

Open
#476 8 comments 0 reactions 2 assignees Claimed by @zhangjyr View on GitHub
area/autoscaling area/gateway kind/feature
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

The autoscaler should support scaling down to 0. When a new request arrives, we should have an activator component intercepts the request and initializes a new pod. Right now, we will simply get the following error if the number of replicas is 0 for a model inference request.

```
error on getting pods for model llama2-7b
```

### Use Case

_No response_

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.