vllm-project / vllm-project/aibrix

Gpu optimizer write deployment replica suggestion and autoscaler go through the calculation again

Open
#461 5 comments 0 reactions 0 assignees View on GitHub
area/heterogeneous kind/bug priority/critical-urgent
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

```
metricsSources:
- endpoint: gpu-optimizer.aibrix-system.svc.cluster.local:8080
path: /metrics/aibrix-system/simulator-llama2-7b-a100
metric: "vllm:deployment_replicas"
targetValue: "1"
```

In heterogeneous story, `gpu_optimizer` expose an endpoint `/metrics/${namespace}/${scale_target_name}`. Seem here're some issues, we used to fetch metrics from pods and run through some calculation to get a desired replica. Here, due to the current component design, `gpu_optimizer` returns a value that autoscaler should adopt. but this would be a different workflow comparing to traditional metrics. The autoscaler will calculate the values and compare with `targetValue` to come up a new value.

Let's double check the logics here.

### Use Case

_No response_

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.