vllm-project / vllm-project/afd-plugin
[Feature]: Move shared-expert computation to the FFN side for NPU Async AFD
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 228
- Forks
- 48
- Avg merge
- 22h 23m
- Merged PRs (30d)
- 44
Description
Feature, motivation, and pitch
Place shared-expert computation on the FFN side for Ascend NPU Async AFD. This aligns shared and routed expert computation with the Attention/FFN split and aims to reduce compute and weight-memory pressure on the Attention side.
Proposed change
- Load and execute shared experts on the FFN side for the targeted NPU Async model paths.
- Carry the required inputs and metadata through the Async connector and combine shared-expert and routed-expert outputs with the model's original semantics.
- Preserve correct stream/event ordering and tensor lifetimes across asynchronous execution.
- Document supported models, topologies, and configuration. Audit existing model-specific handling and reuse it where applicable.
vLLM compatibility and extension points
Target the repository-pinned vLLM/vLLM-Ascend runtime pair. Implement through plugin-owned NPU model, runner, and connector extension points, reusing native operators where possible. Any necessary compatibility patch must be minimal, version-pinned, and documented with an upstream/removal plan; do not modify upstream source trees.
Alternatives considered
Keep shared-expert computation on the Attention side for paths that currently use that placement. This avoids migration work but retains the associated Attention-side compute and memory pressure.
Validation plan
- Compare MoE outputs and end-to-end accuracy against a reference preserving the original shared-expert semantics.
- Cover token partitioning, padding, zero-token ranks, repeated requests, and supported parallel topologies.
- Verify Async communication ordering, output combination, and shutdown on Ascend hardware.
- Measure Attention/FFN memory use, latency, and throughput before and after the change.
Related: #210 (zero shared-expert token handling), #287 (NPU Async prefill).
Before submitting
- I searched existing issues and RFCs.
- I identified whether this belongs in plugin-owned code, compat helpers, or compat patches.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the plugin-owned NPU model, runner, and Async connector extension points, then review related issues #210 and #287 for existing handling. Define the supported model paths and topologies before implementing the migration; done includes reference-accuracy checks, edge-case and ordering validation on Ascend hardware, and before/after memory, latency, and throughput measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100