vllm-project / vllm-project/aibrix

Support per-pod-group-index template differentiation in StormService for multi-node inference

Open
#2,116 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

When deploying vLLM with Pipeline Parallelism (PP) across multiple nodes using StormService, there is a fundamental design gap: **it is impossible to differentiate head and worker pod configurations (labels, probes) while maintaining cross-node communication within the same PodGroup.**

Currently, StormService offers two approaches for multi-node PP deployment, but both have critical limitations:

#### Approach A: Single role with `podGroupSize > 1`

```yaml
roles:
- name: inference
replicas: 1
podGroupSize: 2 # 2 pods form one inference instance
template:
meta
labels:
model.aibrix.ai/name: my-model # Applied to ALL pods
model.aibrix.ai/port: "8000"
spec:
containers:
- name: vllm
readinessProbe: # Applied to ALL pods
httpGet:
path: /health
port: 8000
```

**Problems:**
- ❌ Worker pods (running `--headless`) get `model.aibrix.ai/name` label → Gateway routes requests to worker → **request failures**
- ❌ Worker pods get `readinessProbe` on `/health` → headless mode has no API server → **pods stay NotReady forever** (or no probe at all, losing health checking for head)
- ✅ `PODSET_NAME` is shared across the PodGroup → pods can discover each other
- ✅ Same template → atomic update coordination

#### Approach B: Separate head/worker roles

```yaml
roles:
- name: head
replicas: 1
podGroupSize: 1
template:
meta
labels:
model.aibrix.ai/name: my-model # Only on head ✅
spec:
containers:
- readinessProbe: ... # Only on head ✅

- name: worker
replicas: 1
podGroupSize: 1
template:
metadata:
labels:
model.aibrix.ai/engine: vllm # No model label ✅
spec:
containers: ... # No probe ✅
```

**Problems:**
- ❌ Each role has its own PodSet with different `PODSET_NAME` and `ROLE_TEMPLATE_HASH` → **worker cannot discover head's address** (constructing `${PODSET_NAME}-0.${STORM_SERVICE_NAME}...` points to worker itself)
- ❌ Roles update independently → **no coordinated rolling/in-place update** (updating head breaks worker's connection; updating worker may connect to wrong head)
- ✅ Labels and probes are fully independent per role

### The Core Problem

StormService conflates "pod template" with "pod group", making it impossible to have:
1. Different labels/probes per pod within a PodGroup (head vs worker)
2. Cross-pod communication within the group (shared `PODSET_NAME`)
3. Coordinated updates (atomic template change)

...all at the same time.

### Use Case

- vLLM Pipeline Parallelism: `--tensor-parallel-size 1 --pipeline-parallel-size 2 --nnodes 2`
- Head pod (rank 0): serves OpenAI API, needs `model.aibrix.ai/name` label and `readinessProbe`
- Worker pod (rank 1): runs `--headless`, no API server, should NOT be routed to or probed

Contributor guide

Open the contributing guide

Research direction

The issue does not name specific files, tests, or entry points. Start by locating StormService's pod-group and role-template handling, then trace how PODSET_NAME, ROLE_TEMPLATE_HASH, and rolling updates are assigned; done should support per-index labels and probes while preserving shared discovery and coordinated updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.