vllm-project / vllm-project/aibrix

Add InferencePool Integration Support to Gateway Plugin

Open
#1,233 2 comments 1 reaction 1 assignee Claimed by @Xunzhuo View on GitHub
area/gateway
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

Currently, we patch the configuration with EPP manually and static, see https://github.com/vllm-project/aibrix/blob/main/config/gateway/gateway.yaml#L100

Here is the summary of this proposal:

1. v1 (currently): use envoy gateway EnvoyPatchPolicy (patch original cluster config) + EnvoyExtensionPolicy (add epp ext-proc config to gateway), this is static and not easy to maintain or orchestrate or extensible.
2. v2 (planning): use envoy gateway btp (add host override lb policy) + eep (add epp ext-proc config to gateway, also add receiving/forwarding ns with 'envoy.lb' if needed), this can largely improve UX, and also add fallback abilities to the AIBrix GW.
3. v3 (after v2): use aibrix controller to automatically do what we configure manually in v2, and support InferencePool API. Simplify UX and can adopt GIE conformance test.

### Use Case

1. Simplify configuration and improve UX.
2. Support Fallback and improve reliabilities.
3. Integration with GIE and inference upstream conformance tests.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.