vllm-project / vllm-project/aibrix
Cornserve-inspired Any-to-Any serving: ModelGraph + planner-driven disaggregation + request-static routing
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 697
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 104
Description
### 🚀 Feature Description and Motivation
AIBrix today is strong for text LLM serving (and increasingly disaggregated serving like P/D), but “Any-to-Any” multimodal models (text/image/video/audio in, and text/image/audio out) introduce:
- request-type heterogeneity (different modalities per request)
- computation-path heterogeneity (different components invoked per request)
- compute scaling heterogeneity (encoder/LLM/generator have very different throughput + scaling needs)
Cornserve (paper: arXiv 2512.14098) proposes a practical architecture: model developers describe computation paths; a planner decides colocate vs disaggregate and replica counts; runtime uses request-static (source) routing driven by planner path probabilities.
This issue proposes a concrete way to land these ideas in AIBrix with an incremental MVP.
#### Goals
- Support multi-component model serving (encoder + LLM + generator/vocoder/DiT, etc.) with multiple request types
- Add a declarative model graph (components + deployment options + valid paths)
- Implement request-static routing in AIBrix Gateway using planner-provided path probabilities
- Add a planner/profiler loop to compute replica counts + path splits from workload mix and component throughput
#### Non-goals (for MVP)
- Running arbitrary user-provided Python “composite task” code in the gateway (Cornserve’s record/replay). We’ll start with a declarative graph.
- Perfect optimal planning (Cornserve’s full solver + cell abstraction). Start with a heuristic/ILP-lite planner that is correct and measurable.
paper reference: https://arxiv.org/html/2512.14098v1
### Use Case
N/A
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.