vllm-project / vllm-project/afd-plugin
[Feature]: Add DBO support for Ascend NPU ModelRunnerV2
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 228
- Forks
- 48
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 53
Description
Feature, motivation, and pitch
Ascend NPU ModelRunnerV2 support landed in #257, but the current path explicitly rejects enable_dbo=true and use_ubatching=true. This leaves DBO workloads on ModelRunnerV1 and prevents the Ascend ModelRunnerV2 path from reaching the DBO milestone described in Phase 3 of #240.
Add native DBO support to AFDNPUAttentionModelRunnerV2 for the synchronous CAMP2pAFDConnector path. The implementation should preserve native vLLM/vLLM-Ascend ownership of ModelRunnerV2 request state, input preparation, per-uBatch Attention metadata, output merging, NPU streams/events, and ACL graph lifecycle.
Related work:
- #240 — Ascend NPU ModelRunnerV2 RFC, Phase 3
- #257 — Ascend ModelRunnerV2 implementation without DBO
Proposed change
- Enable
enable_dbo/ native ubatching for the Ascend ModelRunnerV2 Attention path. - Prefer an
AscendUBatchRunnerowned by vLLM-Ascend, exposed through a narrow factory or extension hook. - Integrate AFD transaction, stage, token-shape, DP, and connector-control metadata per uBatch without copying the ModelRunnerV2 execution loop.
- Initially support exactly two uBatches with synchronous
CAMP2pAFDConnector. - Deliver eager DBO first; qualify
FULL_DECODE_ONLYDBO only after eager correctness is established. - Keep ModelRunnerV1 DBO and ModelRunnerV2 no-DBO behavior unchanged.
- Fail closed for unsupported connectors, graph modes, topologies, or uBatch counts.
vLLM compatibility and extension points
Preferred extension point:
- A vLLM-Ascend-owned
AscendUBatchRunnerand an overridable uBatch-runner construction hook in the native NPU ModelRunnerV2 path. - Narrow AFD composition/hooks for connector lifecycle, control publication, AFD forward-context metadata, profiling, and cleanup.
Compat shim needed:
- A version-pinned shim may be used if the pinned vLLM-Ascend runtime does not yet expose the required factory or lifecycle hooks.
Monkey patch needed:
- Avoid if possible. If unavoidable, keep it exact-version, minimal, and documented with an upstream/removal plan.
- Do not copy native
execute_model, input preparation, Attention metadata construction, output merging, or ACL graph-manager lifecycle into afd-plugin.
The implementation targets the repository's pinned vLLM/vLLM-Ascend runtime pair and must not modify either upstream source tree from this repository.
Alternatives considered
- Keep DBO on ModelRunnerV1 only: lowest implementation risk, but leaves Ascend MRV2 feature parity incomplete.
- Port the existing NPU V1 DBO loop into ModelRunnerV2: rejected because it duplicates upstream execution ownership and creates a large upgrade surface.
- Copy the GPU
UBatchRunnerinto afd-plugin: rejected because CUDA-specific stream/graph assumptions do not define the Ascend runtime contract. - Implement eager and ACL graph DBO together: increases the debugging surface; eager-first provides a smaller correctness gate.
Validation plan
Focused tests:
- Configuration accepts the supported two-uBatch MRV2 DBO setup and rejects unsupported combinations before connector rendezvous or weight loading.
- Per-uBatch AFD metadata preserves real and padded token lengths, stage identity, transaction identity, and DP rank agreement.
- Connector control is published before each uBatch data transfer and outside ACL graph bodies.
- Failure paths clean up sibling execution, streams/events, forward contexts, graph state, and connector transactions.
Ascend hardware E2E:
- DeepSeek-V2-Lite, synchronous
CAMP2pAFDConnector, 1A1F, DP1/TP1, eager DBO. - Decode, prefill, mixed batches, uneven splits, padding-only tails, repeated requests, profiling, and clean shutdown.
- Deterministic output comparison against approved native/no-DBO or ModelRunnerV1 references.
- DP2/TP1 and TP2/DP1 coverage before broad topology claims.
- Full non-accuracy NPU E2E gate, then GSM8K-7 accuracy.
- After eager acceptance,
FULL_DECODE_ONLYcapture/replay, graph-key separation by uBatch count/shape, graph misses, and shape changes.
Acceptance criteria:
- Exactly two uBatches execute correctly through native Ascend ModelRunnerV2 DBO ownership.
- No copy of the native MRV2 execution loop or Ascend Attention/graph internals is introduced.
- AFD connector payloads and control metadata are correct for every uBatch.
- Eager correctness, cleanup, and accuracy gates pass on Ascend hardware.
- Graph DBO is not claimed until its separate capture/replay gate passes.
- Existing Ascend MRV2 no-DBO and ModelRunnerV1 DBO tests remain green.
Before submitting
- I searched existing issues and RFCs.
- I identified whether this belongs in plugin-owned code, compat helpers, or compat patches.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the AFDNPUAttentionModelRunnerV2 path and the pinned native NPU ModelRunnerV2 extension points; the issue does not name repository files or test paths. Review the focused configuration, per-uBatch metadata, connector-control, and cleanup tests described in the validation plan. Done means exactly two synchronous CAMP2pAFDConnector uBatches pass eager hardware validation without duplicating native execution ownership, while unsupported combinations fail closed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100