vllm-project / vllm-project/aibrix
[Proposal]: Integration With Envoy AI Gateway
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 697
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 104
Description
### 🚀 Feature Description and Motivation
### Summary
Currently, we deploy Envoy Gateway + Plugin(ext-proc) with a couple of configurations like EnvoyPatchPolicy, EnvoyExtensionPolicy etc. Our vision is focus on inference optimization. But in production scenario, we usually need an unified AI Gateway, which can work as the AI Inference Gateway as well as the AI Unified Gateway.
This will bring duplicated feature implementations between AIBrix Inference GW and Envoy AI GW, and bring extra efforts on traffic management area.
This migration will let AIBrix focus on routing algorithm, performance and inference optimization. And the upstream Envoy AI Gateway work on the integration with plugin ext-proc, provide high level traffic management, service government, security AuthZ/AuthN, extensible ecosystem(MCP/Agent).
This proposal is going to track the process to migrate current architecture into Envoy AI Gateway. Empower the inference Gateway into an unified AI Gateway, here are some advantages, expand the envoy gateway from inference gateway into an AI Gateway, including:
1. Advanced token-based ratelimit: including input/output/total/CEL token cost limit.
2. Unified OpenAI API compatible: chat, embeddings etc.
3. Easier integration with custom Backend (unlike what we manually patched the envoy xds config), we can specify which route should go with ext-proc, which is not
4. Hybrid routing strategy: proxying between in-cluster inference GW and out-cluster Provider, typical scenario is like: routing to a model and fallback or traffic splict between the in-cluster and out-cluster inference service.
5. Advanced Upstream Authentication: support various authentication like API Key, AWS Bedrock, Azure Credentials, and short-lived, long-lived access credentials management.
6. AI/LLM Observability: besides what we exposed in plugin ext-proc, AI Gateway provides unified metrics to access LLM Workloads.
### Goals
1. provide a seamless way for migrating current architecture into Envoy AI Gateway.
2. make sure this migration do not break anything we implemented in AIBrix Plugin.
### Status
Currently, there are some prerequisites for this migration, this will move forward after it is done.
### Use Case
Unified AI/LLM traffic management
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.