vllm-project / vllm-project/aibrix

Feature Request: Network Topology Aware Scheduling Support in AIBrix

Open
#1,730 3 comments 0 reactions 0 assignees View on GitHub
area/orchestration area/scheduling
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

### 🚀 Feature Description and Motivation

In LLM inference scenarios, model parallelism and distributed inference have extremely high requirements for network throughput to exchange data, making networking a critical bottleneck. Modern datacenters have diverse network architectures (e.g., IB, RoCE, NVSwitch) with multiple levels of switches having different throughput and latency characteristics.

**Network Topology Aware Scheduling** allows workloads to be scheduled to the best performance domain with the highest throughput and lowest latency, accelerating data exchange for inference operations.

Therefore, I'm proposing adding **Network Topology Aware Scheduling** support to AIBrix by integrating with Volcano's network topology aware scheduling features: https://volcano.sh/en/docs/network_topology_aware_scheduling/. This will enable LLM inference workloads to be scheduled within optimal network performance domains, significantly improving inference performance.

### Use Case

Consider a datacenter with 8 GPU nodes under a 3-tier switch hierarchy:
Image
S0 to S6 can be considered as switches. If the scheduler is unaware of the underlying network topology, in a 1P1D scenario, the worst-case scenario is that one is on node0 and the other on node7. This requires traversing multiple layers of switches, resulting in long data exchange links and a decrease in inference performance. The best case scenario is that 1P1D is scheduled to the leaf switch, e.g., under S0.

### Proposed Solution

Volcano already supports network topology-aware scheduling, and the Kubeflow community also supports configuring training jobs with network topology-aware scheduling.
- https://www.kubeflow.org/docs/components/trainer/operator-guides/job-scheduling/volcano/
- https://volcano.sh/en/docs/network_topology_aware_scheduling/

Therefore, I think StormService could also support network topology configuration, either by adding a custom field or by configuring network topology requirements in StormService annotations. Then, the StormService Controller could automatically create a Volcano PodGroup for the StormService and inherit the StormService's network topology constraints to the PodGroup, allowing the Volcano scheduler to schedule the PodGroup. This would enable Prefill and Decode pods to be scheduled to the more efficient network performance domains.

Contributor guide

Open the contributing guide

Research direction

Start with the StormService Controller and the linked Volcano network topology-aware scheduling documentation. Trace how StormService configuration reaches the scheduler and how Volcano PodGroups are created or configured. Done means Prefill and Decode pods inherit the requested topology constraints and can be scheduled within the intended network performance domain.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.