Support DeepSeek-VL2 models with MoE and MLA in ExecuTorch

🚀 The feature, motivation and pitch

DeepSeek recently released their Mixture-of-Experts Vision-Language Models, DeepSeek-VL2

DeepSeek-VL2-Tiny, DeepSeek-VL2-Small and DeepSeek-VL2, with 1.0B, 2.8B and 4.5B activated parameters respectively.

The Tiny and Small versions are suitable for on-device usage. The unique features of this model include:

• Mixture-of-Experts (MoE)
• Multi-head Latent Attention (MLA) mechanism for KV Cache efficiency
• Multimodal image understanding

Alternatives

There are distilled reasoning models for DeepSeek R1 (it's mentioned in this https://github.com/pytorch/executorch/issues/7981). However, those models use the same architecture as the targeting models (Llama and Qwen). They don't have the three unique features mentioned above. Especially, MoE and MLA look promising for on-device inference efficiency.

Additional context

No response

RFC (Optional)

Suggested process:

• Leverage the existing llama_transformer, with a set of ExecuTorch infra built around it, like export_llama(https://github.com/pytorch/executorch/blob/main/examples/models/llama/export_llama.py) for export, quantization and lowering to backends.
• MoE: there's an initial version of MoE definition. Feel free to use and extend it.
• MLA: subclass/register the MLA definition of this abstract class, so that it can be re-used and further optimized in future.
• The DeepSeek-VL2 logic and be implemented in its own model.py, in a specific model folder. A reference example is llava model, with similar multimodal structure.

cc @mergennachin @cccclai @helunwencser @dvorjackz

Start by reading examples/models/llama/llama_transformer.py, export_llama.py, and attention.py, then compare the multimodal structure in examples/models/llava. Determine how DeepSeek-VL2's MoE, MLA, and image-understanding logic fit into a new model folder. Done means the Tiny and Small models are supported through ExecuTorch export, quantization, backend lowering, and on-device inference.