vllm-project / vllm-project/aibrix
Providing Dynamic CRD to configure Inferece GW
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
Currently, we exposed a lof of parameters through environment vars, which make the configuration be static, each time we change config, we need to restart the plugin server, which breaks the traffic ext-call.
Also, we can specify the routing strategy for specify model, weight, the criticality of the model etc.
+ PluginConfig: the configuration for dynamic configuring inference GW.
+ PluginControl: the control for dynamic routing for specify models
### Use Case
+ Avaliability improve
+ UX improve
### Proposed Solution
_No response_
Contributor guide
Research direction
Start by tracing the plugin server's environment-variable configuration and the inference gateway's model-routing controls. Clarify the PluginConfig and PluginControl CRD shape, how updates avoid restarting the server, and what dynamic routing behavior and availability guarantees define done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- api, backend-api-design, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100