vllm-project / vllm-project/afd-plugin

[Feature]: Move shared-expert computation to the FFN side for NPU Async AFD

Open
#331 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Ascend feature request
Dominant language
Python
Stars
228
Forks
48
Avg merge
22h 23m
Merged PRs (30d)
44

Description

Feature, motivation, and pitch

Place shared-expert computation on the FFN side for Ascend NPU Async AFD. This aligns shared and routed expert computation with the Attention/FFN split and aims to reduce compute and weight-memory pressure on the Attention side.

Proposed change
  • Load and execute shared experts on the FFN side for the targeted NPU Async model paths.
  • Carry the required inputs and metadata through the Async connector and combine shared-expert and routed-expert outputs with the model's original semantics.
  • Preserve correct stream/event ordering and tensor lifetimes across asynchronous execution.
  • Document supported models, topologies, and configuration. Audit existing model-specific handling and reuse it where applicable.
vLLM compatibility and extension points

Target the repository-pinned vLLM/vLLM-Ascend runtime pair. Implement through plugin-owned NPU model, runner, and connector extension points, reusing native operators where possible. Any necessary compatibility patch must be minimal, version-pinned, and documented with an upstream/removal plan; do not modify upstream source trees.

Alternatives considered

Keep shared-expert computation on the Attention side for paths that currently use that placement. This avoids migration work but retains the associated Attention-side compute and memory pressure.

Validation plan
  • Compare MoE outputs and end-to-end accuracy against a reference preserving the original shared-expert semantics.
  • Cover token partitioning, padding, zero-token ranks, repeated requests, and supported parallel topologies.
  • Verify Async communication ordering, output combination, and shutdown on Ascend hardware.
  • Measure Attention/FFN memory use, latency, and throughput before and after the change.

Related: #210 (zero shared-expert token handling), #287 (NPU Async prefill).

Before submitting
  • I searched existing issues and RFCs.
  • I identified whether this belongs in plugin-owned code, compat helpers, or compat patches.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the plugin-owned NPU model, runner, and Async connector extension points, then review related issues #210 and #287 for existing handling. Define the supported model paths and topologies before implementing the migration; done includes reference-accuracy checks, edge-case and ordering validation on Ascend hardware, and before/after memory, latency, and throughput measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.