vllm-project / vllm-project/aibrix
[Docs]: Enable Heterogeneous SLO Routing & GPU Optimizer for Ascend NPUs (910B / 910C)
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### Summary
I've successfully deployed AIBrix on same-generation Ascend NPUs using the vllm-ascend plugin, and it works perfectly.
However, my goal is to manage a heterogeneous NPU cluster containing mixed generations of Ascend chips (specifically 910B and the newer 910C).
I see that AIBrix currently features a Heterogeneous GPU Optimizer designed to enable cost-efficient mixed-GPU inference with SLO routing. Because I have only deployed on same-gen hardware so far, I haven't deployed any SLO rules or tested the optimizer on NPU yet.
### Motivation
I want to experiment with the GPU Optimizer for my NPU cluster, but I'm unsure if it will work out of the box or if it requires architectural changes. Before I dive into testing, I'd like to ask:
1) Hardware Coupling: Does the current Heterogeneous GPU Optimizer work exclusively with NVIDIA/GPU metrics, or is it hardware-agnostic enough to work with NPUs?
2) Mixed Generations: Can the optimizer differentiate between different generations of the same hardware family (e.g., 910B vs. 910C) the same way it might handle an L20 vs. an A100?
Pointers for Adaptation: If the optimizer does require tweaks to its profiling or gateway metrics to support Ascend NPUs, which parts of the codebase should I focus on?
### Proposed Change
My plan is to deploy the optimizer, test the SLO routing with my 910B/910C hardware, and verify if it behaves correctly. If the optimizer needs to be tweaked to support this heterogeneous NPU scenario, I would love to work on it and deliver a PR upstream.
Any pointers or advice before I start experimenting would be greatly appreciated!
### Alternatives Considered
https://github.com/vllm-project/aibrix/issues/1861
https://github.com/vllm-project/aibrix/issues/1904
There are some issues where some similar scenarios are considered, #1904 was particularly useful for deploy, but I still couldn't find answers to my questions.
Contributor guide
Research direction
Start with the Heterogeneous GPU Optimizer, its profiling and gateway metrics, and the vllm-ascend deployment described here; compare the 910B/910C experiment with issues #1861 and #1904. Done means establishing whether existing SLO routing is hardware-agnostic and, if not, documenting the required changes and validating mixed-generation NPU behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ai-infra-agents, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100