vllm-project / vllm-project/aibrix

[Proposal]: Integration With Envoy AI Gateway

Open
#1,225 3 comments 0 reactions 1 assignee Claimed by @Xunzhuo View on GitHub
area/gateway kind/enhancement priority/important-soon
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

### 🚀 Feature Description and Motivation

### Summary

Currently, we deploy Envoy Gateway + Plugin(ext-proc) with a couple of configurations like EnvoyPatchPolicy, EnvoyExtensionPolicy etc. Our vision is focus on inference optimization. But in production scenario, we usually need an unified AI Gateway, which can work as the AI Inference Gateway as well as the AI Unified Gateway.

Image

This will bring duplicated feature implementations between AIBrix Inference GW and Envoy AI GW, and bring extra efforts on traffic management area.

This migration will let AIBrix focus on routing algorithm, performance and inference optimization. And the upstream Envoy AI Gateway work on the integration with plugin ext-proc, provide high level traffic management, service government, security AuthZ/AuthN, extensible ecosystem(MCP/Agent).

This proposal is going to track the process to migrate current architecture into Envoy AI Gateway. Empower the inference Gateway into an unified AI Gateway, here are some advantages, expand the envoy gateway from inference gateway into an AI Gateway, including:

1. Advanced token-based ratelimit: including input/output/total/CEL token cost limit.
2. Unified OpenAI API compatible: chat, embeddings etc.
3. Easier integration with custom Backend (unlike what we manually patched the envoy xds config), we can specify which route should go with ext-proc, which is not
4. Hybrid routing strategy: proxying between in-cluster inference GW and out-cluster Provider, typical scenario is like: routing to a model and fallback or traffic splict between the in-cluster and out-cluster inference service.
5. Advanced Upstream Authentication: support various authentication like API Key, AWS Bedrock, Azure Credentials, and short-lived, long-lived access credentials management.
6. AI/LLM Observability: besides what we exposed in plugin ext-proc, AI Gateway provides unified metrics to access LLM Workloads.

### Goals

1. provide a seamless way for migrating current architecture into Envoy AI Gateway.
2. make sure this migration do not break anything we implemented in AIBrix Plugin.

### Status

Currently, there are some prerequisites for this migration, this will move forward after it is done.

### Use Case

Unified AI/LLM traffic management

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.