[Feature] DeepSeek-V4 MoE is Blackwell-only via DeepGEMM — is a non-Blackwell path wanted?

### Motivation

DeepSeek-V4 currently reaches only one MoE path in the PyTorch engine: `DeepseekV4MoE` constructs `FusedMoEV4FP4` unconditionally (`lmdeploy/pytorch/models/deepseek_v4.py:633`), which resolves to `TritonFusedMoEV4FP4Builder` and ultimately to DeepGEMM's `m_grouped_fp8_fp4_gemm_nt_contiguous`. That kernel accepts packed FP4 expert weights only on `arch_major == 10`, as the implementation's own docstring notes — so the model is effectively Blackwell-only, and even SM90 is excluded.

That is a reasonable place to start, but it leaves out the hardware most self-hosted deployments in this ecosystem actually have. It also sits oddly next to LMDeploy's own position on old GPUs: TurboMind's MXFP4 support is advertised as working "on NVIDIA GPUs starting from V100". The model file itself carries no architecture gate; the restriction lives entirely in which MoE implementation gets selected.

I would like to know whether a non-Blackwell MoE path for DeepSeek-V4 is something the project wants. If it is, I am in a position to help with the part that is usually hardest to arrange — real hardware and measurements.

### What I have

I have DeepSeek-V4-Flash-0731 serving on 8x A800 (SM80) through a different engine, using the **stock checkpoint with no conversion step** — the published e2m1 weights and e8m0 block scales are read as-is, decoded by a Marlin W4A16 kernel instead of DeepGEMM:

- **GSM8K 96.44%** (1272/1319, greedy, thinking off). The DeepSeek official API scores 96.29% on the same questions and grader, at the same 129 tokens/question — so the serving path costs nothing measurable in quality.
- 105.7 tok/s single-stream with speculative decoding; 330 tok/s aggregate at 16 concurrent
- needle-in-a-haystack clean across 4K / 32K / 128K x three needle depths
- prefill 19 s at 64K, 64 s at 218K; a repeat of the same 218K prompt returns in 0.87 s on a prefix-cache hit
- 1M context ceiling, fp8 KV cache

The point I'd draw from that: a Marlin-family W4A16 GEMM reproduces the published model on Ampere with no accuracy left on the table. There is nothing exotic needed for these weights.

The machine, in case it is useful to know what a contribution here would be validated on — 8x **A800-SXM4-80GB**, NVLink fully connected (NV8 between every pair), driver 590.44.01, 640 GB of GPU memory in one node. It is a dedicated box, not a shared slice, so I can hold a model resident and run whatever matrix is useful rather than working around a queue. If the answer to the question below is yes, testing on real SM80 silicon is the part I can cover — that is usually the awkward half of supporting hardware the maintainers do not have.

The relevant finding for LMDeploy: **DeepSeek's published FP4 experts do not need Blackwell.** A Marlin-family W4A16 GEMM decodes e2m1 with plain bit manipulation and runs on SM80 upward. The Blackwell requirement comes from DeepGEMM specifically, not from the weight format.

### The gap, as far as I can see

`lmdeploy/pytorch/kernels/cuda/` has `fused_moe`, `blocked_fp8`, `w8a8` and `v4_fp4`, but no Marlin-family kernel, so there is currently nothing to fall back to for packed 4-bit experts. The two options I can see:

1. **Dequantise the FP4 experts to BF16 at load and use the generic fused MoE.** Small change, but the expert weights grow roughly 4x — for V4-Flash that does not fit on 8x80GB, so it is not a real option at this scale.
2. **Bring in a Marlin-style W4A16 grouped GEMM** and select it when the FP4 path is unavailable. This is the one that actually works, and it is a substantial piece of kernel work.

### What I am asking

Is (2) something LMDeploy wants? I am not asking anyone to take on Ampere maintenance — I am asking whether a contribution in that direction would be welcome, before I spend days on a kernel port that might not have a home. If it is welcome I would rather do it in the open with the maintainers' guidance on where the pieces belong, and I can carry the A800 testing.

If the answer is that DeepSeek-V4 is intentionally Blackwell-only for now, that is a legitimate scope decision and I would rather hear it plainly than guess from the code.

### Related resources

- `lmdeploy/pytorch/models/deepseek_v4.py:633` — unconditional `FusedMoEV4FP4`
- `lmdeploy/pytorch/backends/cuda/moe/v4_fp4.py:106` — "DeepGEMM's grouped FP8xFP4 kernels accept packed FP4 expert weights only on arch_major == 10"
- #4658 — TurboMind support for the same model (different backend, related question)