[RFC]: DeepSeek V3.2 prefill performance study and token-balanced TP/SP dual batching

Field	Value
Status	Draft
Model	DeepSeek V3.2 W8A8
Primary stage	Prefill
Primary backend	Ascend NPU
Feedback period	One week

Motivation

The current DeepSeek V3.2 CAM async recipe demonstrates prefill execution with
CAMAsyncAFDConnector, DP3PCP8 Attention ranks, and EP8 FFN ranks. Its
benchmark uses a reduced ten-layer model, disables chunked prefill and prefix
caching, uses request-boundary async MoE ubatching, and forces balanced expert
routing.

That setup is useful as a functional and controlled performance baseline, but
it does not answer several questions needed for production sizing:

• Request-boundary dual batching can produce strongly imbalanced ubatches when
  prompt lengths differ. TP/SP execution should be able to split by token count
  while preserving request and attention metadata.
• Prefix-cache hits change the amount of scheduled prefill work and may change
  the optimal AFD topology and ubatch split.
• Real workloads have different prompt-length distributions and request
  imbalance. Results from a single fixed or balanced workload may not
  generalize.
• Chunked prefill changes batch composition and overlap opportunities; the
  selected chunk size may interact with TP/SP, async MoE staging, and expert
  communication.
• The reduced-layer and forced-load-balance setup hides full-model effects and
  natural expert skew.

This RFC proposes one focused implementation change—token-balanced dual-batch
splitting for TP/SP prefill—and a reproducible experiment plan to characterize
these interactions. It does not assume that one configuration will be optimal
for every workload.

Goals

• Implement a two-ubatch token split for supported TP/SP prefill execution.
• Preserve correctness when the split falls inside a request.
• Quantify the effect of controlled prefix-cache hit ratios.
• Quantify the effect of prompt-length/request imbalance.
• Quantify the effect of enabling chunked prefill and varying chunk size.
• Run both reduced-layer diagnostics and full-layer DeepSeek V3.2 experiments.
• Compare forced-balanced routing with natural, non-forced expert routing.
• Optionally compare AFD with a PP/dynamic CPP deployment under an
  apples-to-apples resource and workload budget.
• Produce enough per-layer and end-to-end evidence to explain performance
  changes instead of reporting only aggregate throughput.

Non-goals

• This RFC does not change decode behavior.
• This RFC does not claim a universal default for chunk size, cache policy, or
  topology.
• This RFC does not introduce a new scheduler or prefix-cache implementation.
• This RFC does not make PP/dynamic CPP comparison a blocker for the primary
  AFD study.
• This RFC does not use forced expert balancing for correctness or
  production-representative conclusions because it changes routed expert IDs
  and model outputs.
• This RFC does not generalize results beyond the recorded model revision,
  quantization, software stack, hardware, topology, and workload.

Proposed change

1. Token-balanced TP/SP dual batching

Add a token split policy for the two-stage prefill path. Given the scheduled
token interval [0, T), the target split is near T / 2, subject to the
alignment and padding constraints required by the pinned TP/SP implementation.
Unlike the current request-boundary policy, the split may fall inside a
request.

The implementation must:

• produce two non-empty token slices whenever dual batching is eligible;
• keep the token-count difference minimal after required alignment;
• derive the request slice for each token slice, allowing the boundary request
  to appear in both ubatches;
• rebuild or slice query start locations, sequence lengths, computed-token
  counts, slot mappings, block tables, positions, and Attention metadata for
  the partial boundary request;
• keep padding separate from real scheduled tokens and attach padded tail work
  only to the appropriate stage;
• preserve TP/SP rank-consistent split points and collective ordering;
• preserve the logical output token order when stage outputs are merged;
• fail closed for unsupported attention builders, graph modes, speculative
  metadata, or topology combinations;
• restore forward context and PCP/SP-related mutable state after both success
  and failure;
• expose the split policy and realized token counts in benchmark evidence.

The initial implementation supports exactly two ubatches. Generalizing to more
stages is out of scope until the two-stage metadata and overlap behavior is
validated.

Required focused tests include:

• one long request split across both ubatches;
• multiple equal and unequal requests;
• a split exactly on a request boundary;
• prefix-hit requests with non-zero computed-token counts;
• chunked requests whose current query is only part of the full prompt;
• TP/SP padding and an empty-tail rejection case;
• rank-consistent split decisions;
• output merge order;
• metadata/state cleanup after an exception;
• negative tests for unsupported builders and modes.

2. Reproducible benchmark harness and evidence

Add a recipe-local benchmark manifest and result schema. Every result must
record:

• AFD commit, exact vLLM/vLLM-Ascend revisions, container digest, driver,
  firmware, CANN, PyTorch, and torch-npu versions;
• exact model/checkpoint revision, tokenizer, W8A8 configuration, and layer
  count;
• node/device type and count, network/fabric, rank mapping, TP/SP/PCP/DP/EP/PP
  topology, and Attention:FFN ratio;
• scheduler, eager/graph, async MoE, chunked-prefill, prefix-cache, and expert
  balancing configuration;
• dataset identity, request count, arrival process, prompt/output lengths,
  random seed, warmup, repetitions, and run order;
• requested and realized prefix-cache hit ratio;
• scheduled real tokens, padded tokens, per-ubatch token counts, and imbalance;
• per-rank/per-expert routed token counts.

Use the same immutable software and hardware stack for cases compared in one
figure or table. Randomize or interleave run order where practical, use a
documented warmup, and report at least three measured repetitions with
dispersion rather than only the best run.

3. Metrics

Collect end-to-end serving metrics:

• prefill throughput in input tokens/s;
• request throughput;
• TTFT p50/p90/p95/p99;
• completed/failed requests and sustained-run stability;
• NPU memory use and, where available, power/utilization.

Collect execution metrics sufficient to explain the result:

• Attention, dense MLP, routed MoE, connector dispatch/combine, and other
  communication time per layer;
• stage overlap, exposed bubble time, and critical-path time;
• per-ubatch real/padded tokens and imbalance ratio;
• prefix-cache lookup time, hit tokens, and newly computed tokens;
• chunk count and scheduled tokens per chunk;
• per-expert and per-rank token load, max/mean, coefficient of variation, and
  straggler time;
• TP/SP and EP collective volume/time;
• Attention-side and FFN-side utilization.

Profiler overhead must be measured separately. Headline throughput and TTFT
should come from non-instrumented runs, while instrumented runs explain the
critical path.

Experiment matrix

The study is staged to avoid an unbounded Cartesian product. Each stage first
selects a stable reference configuration; later stages vary one primary factor
at a time and include a small set of interaction cases where the earlier
result shows a meaningful dependency.

Stage A: split-policy validation

Compare:

1. no async MoE dual batching;
2. current request-boundary two-stage split;
3. proposed token-balanced two-stage split.

Start with the reduced ten-layer model for fast profiling, then repeat the
winning and control cases on the full model. Include:

• balanced prompt lengths;
• a bimodal prompt distribution;
• a heavy-tailed or trace-derived prompt distribution;
• a single long request mixed with several short requests.

Report realized per-ubatch token imbalance and correlate it with overlap and
FFN straggler time.

Stage B: prefix-cache hit ratio

Use a dataset generator that controls shared-prefix length and verifies the
realized cache-hit tokens from runtime metrics. Sweep at least:

• 0%;
• 25%;
• 50%;
• 75%;
• 90% or the highest stable realizable ratio.

Run cold-cache and steady-state phases separately. Keep total logical prompt
tokens visible in the result while also reporting newly scheduled prefill
tokens. Do not infer cache ratio only from input construction.

At minimum, compare request-boundary and token-balanced splitting at the
extreme and midpoint hit ratios to determine whether cache hits change the
preferred split.

Stage C: workload/request imbalance

Use deterministic datasets with the same total logical prompt tokens where
possible:

• uniform/equal-length requests;
• narrow production-like distribution;
• bimodal short/long requests;
• heavy-tailed or trace-derived distribution;
• adversarial ordering with long requests clustered on one side of the batch.

Record mean, percentiles, standard deviation, coefficient of variation, and
max/min prompt length. Repeat with shuffled and fixed ordering so scheduler
ordering is not confused with the distribution itself.

Stage D: chunked prefill

Compare chunked prefill disabled with a power-of-two size sweep selected for
the frozen stack, initially:

• 8K tokens;
• 16K tokens;
• 32K tokens;
• 64K tokens;
• the unchunked reference.

Invalid sizes for the chosen block, TP/SP, graph, or maximum-batched-token
constraints should be omitted with the reason recorded. Keep scheduler
capacity and concurrency explicit. Measure both throughput/TTFT and the
interaction with dual-batch balance, padding, cache hits, and expert load.

After the main sweep, repeat the best two sizes under one balanced and one
imbalanced workload and under low and high prefix-cache hit ratios.

Stage E: full layers and natural expert routing

The final performance conclusions must include the complete DeepSeek V3.2
layer count and natural, non-forced routing.

Use the following diagnostic pairs:

Model depth	Expert routing	Purpose
Reduced ten layers	Forced balanced	Fast mechanism isolation and regression with the existing recipe
Reduced ten layers	Natural routing	Isolate expert-skew effects cheaply
Full layers	Forced balanced	Separate depth scaling from expert skew; performance-only
Full layers	Natural routing	Production-representative primary result

For natural routing, preserve the model router's expert IDs and report
per-layer expert heat and rank skew. Any forced-balanced result must be labeled
synthetic and must not be used for accuracy claims.

Stage F (optional): PP/dynamic CPP comparison

If the pinned stack has a validated PP/dynamic CPP configuration for
DeepSeek V3.2 prefill, compare it with the best stable AFD configuration using:

• the same total NPU count and hardware generation;
• the same full model and checkpoint;
• natural expert routing;
• the same workload, prefix-cache state, chunk policy, warmup, and repetition
  rules;
• topology-specific memory and communication metrics.

Record exact dynamic CPP terminology, flags, constraints, and upstream
revision in the evidence. If parity in device count, supported features, or
correctness cannot be established, report the comparison as exploratory
rather than a winner/loser claim.

Reference configurations

The checked-in DeepSeek V3.2 CAM async recipe is the starting point, not a
permanent benchmark contract:

• baseline: DP4PCP8;
• AFD Attention: DP3PCP8;
• AFD FFN: EP8;
• async MoE stages: 2;
• maximum batched tokens: 140000;
• output length: 1 for prefill isolation;
• current reduced model: first ten layers.

Before measurement begins, the experiment PR must freeze:

• exact physical topology and whether the requested TP/SP terminology maps to
  PCP or another stack-specific mode;
• the full-model launch commands;
• chunk-size flag semantics;
• cache-hit generator and runtime verification;
• the PP/dynamic CPP reference, if included.

Correctness gates

No performance result is accepted until:

• token-balanced and unsplit outputs match within the quantized model's
  documented tolerance for every supported topology;
• prefix-hit output matches the equivalent uncached execution;
• chunked and unchunked outputs match within tolerance;
• the full-layer natural-routing case passes the selected accuracy smoke test;
• repeated mixed-length and partial-request splits show no hangs, collective
  mismatch, metadata leakage, or state contamination;
• unsupported combinations fail before communication resources are created.

Plugin boundary

Plugin-owned:

• async MoE split policy and token-slice selection;
• AFD connector payload slicing and merge ordering;
• benchmark datasets/manifests, instrumentation, recipes, and result schema;
• validation and fail-closed policy for supported AFD mode combinations.

Versioned compatibility helper or patch, only if required:

• exact pinned vLLM/vLLM-Ascend metadata-builder or TP/SP seams that cannot be
  reached through composition or inheritance.

Any patch must follow AGENTS.md: copy the exact pinned upstream signature and
return type, explain the patch immediately above the function, mark every
AFD-only delta, add focused drift tests, assess performance, and document an
upstream/removal plan.

Upstream-owned and unchanged:

• scheduler and prefix-cache policy;
• chunked-prefill scheduling;
• native Attention/MLA, MoE, quantization, TP/SP/PCP, EP, and PP kernels and
  collectives;
• native expert routing in non-forced experiments;
• non-AFD serving behavior.

Staged implementation

1. Freeze the stack, topology terminology, benchmark manifest, and correctness
   tolerances.
2. Add token-balanced split contracts and CPU/unit tests.
3. Implement the TP/SP metadata and connector slicing path with fail-closed
   validation.
4. Run reduced-layer Stage A profiling and select stable reference cases.
5. Run prefix-cache, workload-imbalance, and chunk-size studies.
6. Run full-layer forced-balanced and natural-routing validation.
7. Optionally run the resource-matched PP/dynamic CPP comparison.
8. Publish raw results, summarized tables/plots, profiler methodology, and
   recommended operating regions with limitations.

Implementation and benchmark evidence may land in separate PRs, but the
full-layer natural-routing evidence is required before this RFC makes a
production performance recommendation.

Risks and alternatives

Risks

• Splitting inside a request makes Attention metadata and cache addressing more
  complex than request-boundary splitting.
• TP/SP/PCP terminology and private metadata contracts may differ across the
  current recipe and the eventual pinned upstream stack.
• Prefix hits and chunks can produce very small or empty work in one stage.
• A large experiment matrix can consume substantial NPU time without isolating
  causality.
• Profiler instrumentation can perturb overlap.
• Forced balancing can hide the production bottleneck; natural routing can
  make repeated runs noisier.
• Full-model results may not scale linearly from the ten-layer diagnostic.

Mitigations are fail-closed contracts, staged experiments, immutable manifests,
realized-work metrics, non-profiled headline runs, repeated measurements, and
mandatory full-layer natural-routing evidence.

Alternatives considered

• Keep request-boundary splitting: simpler metadata, but can leave large
  token imbalance for mixed-length workloads.
• Split only on request boundaries after sorting requests: can improve some
  batches but changes scheduler order and still cannot balance a single long
  request.
• Benchmark only the reduced model with forced balancing: fast and stable,
  but does not represent full-layer expert skew.
• Run the complete Cartesian product: exhaustive in theory, but expensive
  and difficult to interpret; the staged matrix focuses interaction runs after
  main effects are measured.
• Make PP/dynamic CPP mandatory: useful for positioning, but it should not
  block validation of the AFD implementation when a comparable upstream
  configuration is unavailable.

Acceptance criteria

• Supported TP/SP prefill batches can use two token-balanced ubatches,
  including a split inside one request.
• Split metadata, cache addressing, collective ordering, and merged outputs are
  covered by focused tests and correctness evidence.
• The benchmark manifest and raw result schema are checked in and reproducible.
• Prefix-cache experiments report requested and realized hit ratios.
• Workload experiments include balanced, bimodal, and heavy-tailed/trace-like
  prompt distributions.
• Chunked prefill is compared with multiple valid chunk sizes and an unchunked
  reference.
• Headline conclusions include full-layer DeepSeek V3.2 with natural expert
  routing.
• Expert imbalance is reported per layer/rank and correlated with the critical
  path.
• Results include both end-to-end and per-layer/communication evidence.
• Optional PP/dynamic CPP results are clearly marked as comparable or
  exploratory.
• Unsupported combinations fail early and are documented.

Feedback requested

1. For the first implementation, which exact TP/SP/PCP topology and pinned
   vLLM-Ascend revision should define the supported contract?
2. What alignment constraints must the token split obey for the selected
   Attention and communication kernels?
3. Is a two-stage split inside one request acceptable, or are there metadata
   builders that must remain request-boundary-only?
4. Which prefix-hit ratios and workload distributions best represent the
   production target?
5. Which chunk sizes should be added to or removed from the initial sweep?
6. Which full-model topology should be the primary natural-routing result?
7. Is there a resource-comparable PP/dynamic CPP setup that should be included,
   and what exact configuration defines it?

Feedback period

Proposed feedback period: one week, through 2026-08-04. During review, work may
begin on the benchmark manifest, synthetic workload generation, and
CPU-testable split contracts. Runtime implementation should begin after the
supported TP/SP topology and pinned stack are agreed.

Related work

• Existing DeepSeek V3.2 CAM async prefill recipe:
  recipe/npu/CAMAsyncAFDConnector/deepseek_v3_2/
• Existing AFD NPU ubatch helpers:
  afd_plugin/v1/worker/npu/ubatch_utils.py
• vLLM-Ascend RFC: Prefill Performance Optimization for DeepSeek Large Scale EP
  (vllm-project/vllm-ascend#3012)

CC

@jiaran-king @specture724 @yujuancao07