vllm-project / vllm-project/aibrix

[RFC] Add model API as the umbrella for model centric deployment

Open
#302 3 comments 1 reaction 1 assignee Claimed by @varungup90 View on GitHub
kind/enhancement kind/feature priority/critical-urgent
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

As we discussed offline, we consider to provide a Model API object, designed to simplify and unify the deployment of models by managing all related configurations under one umbrella. Traditionally, users have had to manually set up components like HTTPRoute, HPA, initContainer, and other runtime specifics. This process requires careful coordination between user and infrastructure, adding complexity to deployment workflows.

With the Model API object, we aim to:

- **Simplify deployment**: Automatically manage all necessary components (such as routing, scaling, and runtime setup) through a single object, reducing user-side configuration overhead.
- **Enable features seamlessly**: Our umbrella object makes it easier to bring features like heterogenous, distributed serving, traffic routing, and runtime initialization to model-centric deployments with minimal user effort.

### Use Case

_No response_

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.