Onboard command-a-plus to MAD (vLLM default tier, MoE VLM)

Requesting team / customer: AMD FDE / Cohere Inference v-team (customer: Cohere — enterprise, on-prem/no-API; this benchmarking work supports Cohere's internal training pipeline evaluation and the longer-term goal of Cohere serving on AMD Instinct).
Jira Epic: pending — AIMODELS project key/access still being confirmed by requester; will link once created per Phase 1 of the intake process.

Model

HF repo: CohereLabs/command-a-plus-05-2026-fp8
Confirmed identity (2026-08-19, via HF Hub API + raw config.json): this is not a dense 111B text model. It is a vision-language MoE: Cohere2VisionForConditionalGeneration wrapping a Cohere2MoeForCausalLM text backbone.

Text backbone (MoE): 128 experts, top-8 routing per token, 4 shared experts, sigmoid expert-selection, 32 hidden layers, hidden_size 4096, 128 attention heads / 8 KV heads (GQA), 200K max context, 262144 vocab. Sliding-window attention pattern (3 local + 1 full, repeating).
Vision tower: SigLIP (27 layers, 512px images, 16px patches).
Total params: ~218.8B (6.2B BF16 + 212.6B F8_E4M3 per safetensors metadata).
Quantization: FP8 via compressed-tensors (float-quantized, channel-wise weight-only + dynamic token-wise activation), quantized from a bf16 base checkpoint (CohereLabs/command-a-plus-05-2026-bf16). The ignore list keeps attention projections, MoE router gates (mlp.gate), the vision tower, multimodal projector, and lm_head at bf16 — only the MoE expert FFN weights are FP8.
License: Apache-2.0.

Framework: vLLM
Proposed MAD tier: vllm_default (full sweep — tp8, mc 1/8/32/128) as a placeholder dense-style TP8 baseline (see open items below — this may not be the most representative config given the confirmed MoE architecture)
Precision: dtype: auto (checkpoint already declares its own FP8/compressed-tensors quantization config; not overridden to float16 like the plain-dense model group in default.yaml)

Hardware target
AMD Instinct MI300X (gfx942)
Workload / win-criteria
Standard MAD serving benchmark sweep (throughput/latency vs concurrency), used to validate Cohere's MoE training checkpoint runs correctly and performantly on MI300X via vLLM.
Proposed changes

scripts/vllm/models.json — 1 new entry (pyt_vllm_command-a-plus_fp8)
scripts/vllm/configs/default.yaml — 1 new entry (command-a-plus)

Draft config content available on request; no PR opened yet pending sign-off from requester and confirmation of Jira Epic link.
Open items before Phase 3 (Scoping) can close

Confirm "command-a-plus" naming/identity with Cohere/requester — RESOLVED 2026-08-19: CohereLabs/command-a-plus-05-2026-fp8 (requester-supplied HF URL), confirmed via HF Hub API + config.json.
command-a-plus's text backbone is a genuine 128-expert/top-8 MoE (Cohere2MoeForCausalLM), architecturally similar in kind (not scale) to a WideEP-style expert-parallel problem. Decide whether initial MAD onboarding should be a plain co-located TP8 vllm_default sweep (current draft) or whether an expert-parallel-aware config should be considered from the start. Needs requester/Rahul/Cohere input.
command-a-plus is vision-language (Cohere2VisionForConditionalGeneration, pipeline_tag image-text-to-text). Current draft benchmarks it text-only (no image inputs). Confirm whether multimodal benchmarking is in scope for this round of onboarding or explicitly deferred.
Jira https://amd-hub.atlassian.net/browse/AIMODELS-1341

Filed per AMD's AI Models Intake Process (Confluence: MLSE space, "AI Models Intake Process"), Phase 2 "Technical Intake." This is 1 of 3 companion issues (command-r7b / command-a-plus / GLM-5.1-FP8) covering the same customer engagement.

Start by reviewing existing model entries in scripts/vllm/models.json and benchmark tiers in scripts/vllm/configs/default.yaml, then compare the proposed command-a-plus configuration with nearby entries. Before implementation, resolve whether the MoE needs expert-parallel handling and whether multimodal inputs are in scope; done means the agreed configuration is added and the standard MI300X vLLM sweep is defined.