vllm-project / vllm-project/aibrix
Add resource configuration fields for batch inference deployment in UI
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 697
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 104
Description
### 🚀 Feature Description and Motivation
**The model deployment page**
This page currently supports only two resource parameters:
GPU type
GPU count per engine
To enable intelligent resource requests for the batch inference scenario, we'd like to add the following two required fields to this page:
1. replica: The number of inference instances that will be created in the batch inference.
2. configuration strategy: Determines whether the selected GPU settings and replica count can be modified. Two options are supported:
• Preferred — values serve as a recommendation and may be adjusted.
• Required — values are fixed and cannot be modified.
### Use Case
batch
### Proposed Solution
The replica field can be added into the accelerator part.
The configuration strategy filed can be added into into the accelerator part or as a global options to overwrite other deployment specs as well.
Contributor guide
Assessment
This issue has not been assessed yet.