vllm-project / vllm-project/aibrix

Map gateway SLO tiers to vLLM request priority

Open
#2,286 1 comment 0 reactions 0 assignees View on GitHub
area/gateway kind/feature
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### Feature Description and Motivation

We rank and queue requests by SLO at the gateway (`slo.go`, `queue/slo_queue.go`), but once dispatched the engine schedules FCFS, so that ordering doesn't survive into vLLM. vLLM supports `--scheduling-policy priority` with a per-request `priority`. Passing our SLO tier through as priority lets the engine preempt/order in line with SLO, so a latency-sensitive request that lands behind a batch of low-tier ones still runs first inside the engine.

### Use Case

Mixed-tier traffic on shared replicas (interactive vs batch). Today a burst of low-priority work can head-of-line block latency-sensitive requests at the engine, even when the gateway tried to prioritize them.

### Proposed Solution

Gateway derives a `priority` from the request's SLO class and sets it on the upstream vLLM request; deployments run with `scheduling-policy=priority`. Map a small fixed set of tiers and document the semantics (lower value = higher priority in vLLM).

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the SLO class flows through slo.go and queue/slo_queue.go, then locate where the gateway builds the upstream vLLM request. Confirm vLLM's priority semantics and define the fixed tier mapping; the work is done when the priority reaches vLLM, the deployment setting and lower-value-is-higher-priority behavior are documented, and the relevant behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.