vllm-project / vllm-project/aibrix

Using UDS to improve Inference GW plugin latency

Open
#1,200 0 comments 0 reactions 0 assignees View on GitHub
area/gateway
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

### 🚀 Feature Description and Motivation

Currently we deploy plugin(ext-proc) as a deployment in cluster, which communicates with envoy through gprc call in pod-to-pod networking, which brings external cost for different NetNS, node data transfer.

Support deploying plugin as a sidecar of envoyproxy to use UDS communitcation between containers.

### Use Case

System performance improvemnt

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by tracing the current ext-proc plugin deployment and Envoy gRPC communication, then determine how sidecar deployment and UDS communication should be integrated; done means the plugin runs beside Envoy and the latency improvement is demonstrated.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc, kubernetes
Domain
distributed-systems, infrastructure, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.