vllm-project / vllm-project/aibrix
Map gateway SLO tiers to vLLM request priority
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### Feature Description and Motivation
We rank and queue requests by SLO at the gateway (`slo.go`, `queue/slo_queue.go`), but once dispatched the engine schedules FCFS, so that ordering doesn't survive into vLLM. vLLM supports `--scheduling-policy priority` with a per-request `priority`. Passing our SLO tier through as priority lets the engine preempt/order in line with SLO, so a latency-sensitive request that lands behind a batch of low-tier ones still runs first inside the engine.
### Use Case
Mixed-tier traffic on shared replicas (interactive vs batch). Today a burst of low-priority work can head-of-line block latency-sensitive requests at the engine, even when the gateway tried to prioritize them.
### Proposed Solution
Gateway derives a `priority` from the request's SLO class and sets it on the upstream vLLM request; deployments run with `scheduling-policy=priority`. Map a small fixed set of tiers and document the semantics (lower value = higher priority in vLLM).
Contributor guide
Research direction
Start by tracing how the SLO class flows through slo.go and queue/slo_queue.go, then locate where the gateway builds the upstream vLLM request. Confirm vLLM's priority semantics and define the fixed tier mapping; the work is done when the priority reaches vLLM, the deployment setting and lower-value-is-higher-priority behavior are documented, and the relevant behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100