vllm-project / vllm-project/aibrix
Support per-pod-group-index template differentiation in StormService for multi-node inference
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
When deploying vLLM with Pipeline Parallelism (PP) across multiple nodes using StormService, there is a fundamental design gap: **it is impossible to differentiate head and worker pod configurations (labels, probes) while maintaining cross-node communication within the same PodGroup.**
Currently, StormService offers two approaches for multi-node PP deployment, but both have critical limitations:
#### Approach A: Single role with `podGroupSize > 1`
```yaml
roles:
- name: inference
replicas: 1
podGroupSize: 2 # 2 pods form one inference instance
template:
meta
labels:
model.aibrix.ai/name: my-model # Applied to ALL pods
model.aibrix.ai/port: "8000"
spec:
containers:
- name: vllm
readinessProbe: # Applied to ALL pods
httpGet:
path: /health
port: 8000
```
**Problems:**
- ❌ Worker pods (running `--headless`) get `model.aibrix.ai/name` label → Gateway routes requests to worker → **request failures**
- ❌ Worker pods get `readinessProbe` on `/health` → headless mode has no API server → **pods stay NotReady forever** (or no probe at all, losing health checking for head)
- ✅ `PODSET_NAME` is shared across the PodGroup → pods can discover each other
- ✅ Same template → atomic update coordination
#### Approach B: Separate head/worker roles
```yaml
roles:
- name: head
replicas: 1
podGroupSize: 1
template:
meta
labels:
model.aibrix.ai/name: my-model # Only on head ✅
spec:
containers:
- readinessProbe: ... # Only on head ✅
- name: worker
replicas: 1
podGroupSize: 1
template:
metadata:
labels:
model.aibrix.ai/engine: vllm # No model label ✅
spec:
containers: ... # No probe ✅
```
**Problems:**
- ❌ Each role has its own PodSet with different `PODSET_NAME` and `ROLE_TEMPLATE_HASH` → **worker cannot discover head's address** (constructing `${PODSET_NAME}-0.${STORM_SERVICE_NAME}...` points to worker itself)
- ❌ Roles update independently → **no coordinated rolling/in-place update** (updating head breaks worker's connection; updating worker may connect to wrong head)
- ✅ Labels and probes are fully independent per role
### The Core Problem
StormService conflates "pod template" with "pod group", making it impossible to have:
1. Different labels/probes per pod within a PodGroup (head vs worker)
2. Cross-pod communication within the group (shared `PODSET_NAME`)
3. Coordinated updates (atomic template change)
...all at the same time.
### Use Case
- vLLM Pipeline Parallelism: `--tensor-parallel-size 1 --pipeline-parallel-size 2 --nnodes 2`
- Head pod (rank 0): serves OpenAI API, needs `model.aibrix.ai/name` label and `readinessProbe`
- Worker pod (rank 1): runs `--headless`, no API server, should NOT be routed to or probed
Contributor guide
Research direction
The issue does not name specific files, tests, or entry points. Start by locating StormService's pod-group and role-template handling, then trace how PODSET_NAME, ROLE_TEMPLATE_HASH, and rolling updates are assigned; done should support per-index labels and probes while preserving shared discovery and coordinated updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100