vllm-project / vllm-project/aibrix
[Feat] Support Kubernetes-native service discovery in TensorRT-LLM disaggregated router (engine-side)
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
TRT-LLM's native disagg proxy (OpenAIDisaggServer) currently supports three service discovery modes: static URLs in disagg_config.yaml, metadata polling via etcd (-m metadata_config.yaml), and cluster watch via etcd with heartbeat/TTL. None of these are Kubernetes-native — they all require an etcd sidecar or hardcoded URLs
For AIBrix integration, TRT-LLM should support Kubernetes-native service discovery where worker pods are discovered via K8s Service/EndpointSlice resources (using labels like model.aibrix.ai/name and pod annotations for role). This is an engine-side change to the TRT-LLM codebase — adding a K8s discovery backend alongside the existing etcd-based ones. This would eliminate the need for an etcd dependency in K8s-native deployments and align with how AIBrix discovers vLLM/SGLang pods today.
### Use Case
for Kubernetes users.
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.