vllm-project / vllm-project/aibrix

[Feat] Support Kubernetes-native service discovery in TensorRT-LLM disaggregated router (engine-side)

Open
#2,017 0 comments 0 reactions 1 assignee Claimed by @Jeffwan View on GitHub
area/inference-engine kind/feature
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

TRT-LLM's native disagg proxy (OpenAIDisaggServer) currently supports three service discovery modes: static URLs in disagg_config.yaml, metadata polling via etcd (-m metadata_config.yaml), and cluster watch via etcd with heartbeat/TTL. None of these are Kubernetes-native — they all require an etcd sidecar or hardcoded URLs

For AIBrix integration, TRT-LLM should support Kubernetes-native service discovery where worker pods are discovered via K8s Service/EndpointSlice resources (using labels like model.aibrix.ai/name and pod annotations for role). This is an engine-side change to the TRT-LLM codebase — adding a K8s discovery backend alongside the existing etcd-based ones. This would eliminate the need for an etcd dependency in K8s-native deployments and align with how AIBrix discovers vLLM/SGLang pods today.

### Use Case

for Kubernetes users.

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.