vllm-project / vllm-project/aibrix

Cornserve-inspired Any-to-Any serving: ModelGraph + planner-driven disaggregation + request-static routing

Open
#1,868 2 comments 0 reactions 1 assignee Claimed by @Jeffwan View on GitHub
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

### 🚀 Feature Description and Motivation

AIBrix today is strong for text LLM serving (and increasingly disaggregated serving like P/D), but “Any-to-Any” multimodal models (text/image/video/audio in, and text/image/audio out) introduce:

- request-type heterogeneity (different modalities per request)
- computation-path heterogeneity (different components invoked per request)
- compute scaling heterogeneity (encoder/LLM/generator have very different throughput + scaling needs)

Cornserve (paper: arXiv 2512.14098) proposes a practical architecture: model developers describe computation paths; a planner decides colocate vs disaggregate and replica counts; runtime uses request-static (source) routing driven by planner path probabilities.

This issue proposes a concrete way to land these ideas in AIBrix with an incremental MVP.

#### Goals

- Support multi-component model serving (encoder + LLM + generator/vocoder/DiT, etc.) with multiple request types
- Add a declarative model graph (components + deployment options + valid paths)
- Implement request-static routing in AIBrix Gateway using planner-provided path probabilities
- Add a planner/profiler loop to compute replica counts + path splits from workload mix and component throughput

#### Non-goals (for MVP)

- Running arbitrary user-provided Python “composite task” code in the gateway (Cornserve’s record/replay). We’ll start with a declarative graph.
- Perfect optimal planning (Cornserve’s full solver + cell abstraction). Start with a heuristic/ILP-lite planner that is correct and measurable.

paper reference: https://arxiv.org/html/2512.14098v1

### Use Case

N/A

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.