vllm-project / vllm-project/aibrix
[RFC]: AIBrix Multi-Modality: Best-in-Class Omni-Modal Serving Platform
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### Summary
vLLM-Omni already provides the multi-stage serving framework with pluggable connectors, Cache-DiT acceleration, and OpenAI-compatible APIs. AIBrix provides Kubernetes-native inference infrastructure with gateway routing, autoscaling, PrisKV KV cache store, and StormService orchestration.
This proposal integrates them deeply — making AIBrix the best platform to run multi-modal models powered by vLLM-Omni.
### Motivation
Multi-modal AI is moving from research demos to production workloads. Applications now combine text chat, image generation, video generation, speech recognition (ASR), and text-to-speech (TTS) in a single user experience. Serving these workloads efficiently requires solving problems that neither standalone LLM inference nor single-model diffusion serving addresses:
- Heterogeneous pipeline stages — An omni pipeline chains ASR (1.7B params, lightweight) → LLM (235B params, compute-heavy prefill, memory-bound decode) → DiT (burst GPU for diffusion steps) → TTS (real-time audio streaming). Each stage has fundamentally different resource profiles.
- Inter-stage data transfer — KV caches, visual tokens, and audio embeddings must flow between stages with minimal latency. CPU-staged copies are a bottleneck.
- Independent scaling — Image generation traffic spikes don't correlate with text chat load. Scaling the entire pipeline uniformly wastes GPUs.
- GPU cost — A naive deployment dedicates one GPU per model. For a full omni pipeline (ASR + LLM + DiT + TTS), that's at least 4+ GPUs minimum, even when most models are idle most of the time.
- Cloud-native orchestration — Production deployments run on Kubernetes. Ray adds a second distributed runtime on top of K8s, increasing operational complexity.
### Proposed Change
TODO. A proposal will come soon
### Alternatives Considered
_No response_
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by reading AIBrix's existing Kubernetes, gateway, autoscaling, PrisKV, and StormService components alongside vLLM-Omni's serving framework; the promised proposal must first define the integration scope, implementation locations, and measurable completion criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- ai-infra-agents, cloud, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100