vllm-project / vllm-project/aibrix

[Feature]: Split deployment identifier from model deployment and add support for custom port for model deployment

Open
#1,086 4 comments 0 reactions 1 assignee Claimed by @varungup90 View on GitHub
kind/support priority/important-soon
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

I deploy the R1 deployment service with mindie, then I create the HTTPRoute and service for the deployment. The label value `model.aibrix.ai/name: deepseek-r1-w02mfd` is not same to the start command R1-int8
`gateway_req_body.go:60] "model doesn't exist in cache, probably wrong model name" requestID="1832b9c9-1772-42ac-bc4c-cd5572d15097" model="R1-int8"`.
I think it should be possible to access through the model in the header along with the name in the model, rather than just accessing through the gateway plugin based on the name in the model
### Use Case

```
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
creationTimestamp: "2025-05-13T06:36:02Z"
generation: 2
name: deepseek-r1-w02mfd-router
namespace: aibrix-system
resourceVersion: "25629649"
uid: 0572e0d4-8706-45de-b944-4dc927bc7dac
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: aibrix-eg
namespace: aibrix-system
rules:
- backendRefs:
- group: ""
kind: Service
name: deepseek-r1-w02mfd
namespace: prdsafe
port: 1025
weight: 1
matches:
- headers:
- name: model
type: Exact
value: deepseek-r1-w02mfd
path:
type: PathPrefix
value: /v1/completions
- headers:
- name: model
type: Exact
value: deepseek-r1-w02mfd
path:
type: PathPrefix
value: /v1/chat/completions
timeouts:
request: 120s
status:
parents:
- conditions:
- lastTransitionTime: "2025-05-13T07:08:05Z"
message: Route is accepted
observedGeneration: 2
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2025-05-13T07:08:05Z"
message: Resolved all the Object references for the Route
observedGeneration: 2
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: aibrix-eg
namespace: aibrix-system
```
```
Header: model: deepseek-r1-w02mfd
Body:
{
"model": "R1-int8",
"max_tokens":4096,
"temperature":0.6,
"stream": false,
"messages": [
{
"role": "user",
"content": "深入分析调度器的过滤和打分机制,越详细越好"
}
]
}
```
### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.