vllm-project / vllm-project/aibrix

Add resource configuration fields for batch inference deployment in UI

Open
#2,359 2 comments 0 reactions 1 assignee Claimed by @Jeffwan View on GitHub
area/batch
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

### 🚀 Feature Description and Motivation

**The model deployment page**

This page currently supports only two resource parameters:

GPU type
GPU count per engine

To enable intelligent resource requests for the batch inference scenario, we'd like to add the following two required fields to this page:

1. replica: The number of inference instances that will be created in the batch inference.
2. configuration strategy: Determines whether the selected GPU settings and replica count can be modified. Two options are supported:
• Preferred — values serve as a recommendation and may be adjusted.
• Required — values are fixed and cannot be modified.

### Use Case

batch

### Proposed Solution

The replica field can be added into the accelerator part.
The configuration strategy filed can be added into into the accelerator part or as a global options to overwrite other deployment specs as well.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.