vllm-project / vllm-project/aibrix

Autoscaler-driven elastic expert/data parallelism for MoE

Open
#2,288 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/autoscaling area/distributed kind/feature
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

Feature Description and Motivation

For large MoE models we only scale whole replicas, which is coarse and expensive. vLLM has elastic EP (enable_elastic_ep, /is_scaling_elastic_ep, the DP coordinator request-wave protocol in vllm/v1/engine/coordinator.py) to add/remove DP/EP ranks at runtime. The autoscaler doesn't drive it. Scaling EP width by load would right-size MoE serving without full-replica steps.

Use Case

DeepSeek / Qwen-MoE style deployments where per-replica cost is high and traffic varies; scale expert parallelism up and down instead of adding or removing entire replicas.

Proposed Solution

Let PodAutoscaler target EP/DP width for elastic-EP-enabled deployments and drive scaling via the engine's elastic-EP endpoints + the DP coordinator wave protocol. Start observe-only (read /is_scaling_elastic_ep and wave state) before acting on it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the elastic-EP references in vllm/v1/engine/coordinator.py and review the /is_scaling_elastic_ep endpoint and DP coordinator request-wave protocol named in the issue. Trace how PodAutoscaler currently targets replicas, then define how observe-only wave-state reporting and later EP/DP width changes would be validated for elastic-EP-enabled deployments.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
backend, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.