vllm-project / vllm-project/aibrix

LLMPilot: Generate the best deployment configuration for model + GPU combination

Open
#90 6 comments 0 reactions 1 assignee Claimed by @Jeffwan View on GitHub
area/benchmark help wanted priority/critical-urgent
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

For the 33b model deployment, we have a few options, A10, V100-32GiB, L20, L40. Technically, we can launch the instance using M * N GPU Type. However, we need to evaluate the most optimal plan for given latency/throughput/cost/ goals.

Selecting the most optimal GPU deployment for the model is a complex task that requires careful evaluation of those key metrics. By conducting benchmarks, analyzing costs, and considering community input, we can make an informed decision that meets our project goals. This RFC serves as a starting point for the discussion and invites contributions from all stakeholders.

### Use Case

As a user, I want to know the best gpu types to run a specific model.

### Proposed Solution

1. benchmark tools + benchmark datasets (pluggable)
2. experiment plans
3. generate results

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.