vllm-project / vllm-project/aibrix
Gpu optimizer write deployment replica suggestion and autoscaler go through the calculation again
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
```
metricsSources:
- endpoint: gpu-optimizer.aibrix-system.svc.cluster.local:8080
path: /metrics/aibrix-system/simulator-llama2-7b-a100
metric: "vllm:deployment_replicas"
targetValue: "1"
```
In heterogeneous story, `gpu_optimizer` expose an endpoint `/metrics/${namespace}/${scale_target_name}`. Seem here're some issues, we used to fetch metrics from pods and run through some calculation to get a desired replica. Here, due to the current component design, `gpu_optimizer` returns a value that autoscaler should adopt. but this would be a different workflow comparing to traditional metrics. The autoscaler will calculate the values and compare with `targetValue` to come up a new value.
Let's double check the logics here.
### Use Case
_No response_
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.