deepseek-ai / deepseek-ai/DeepSpec

feat: support Multi-Token Prediction (MTP) training for LLM and speech modalities

Open
#70 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
7.1k
Forks
667
PR merge metrics
No merged PRs in 30d

Description

## Feature: Support Multi-Token Prediction (MTP) Training for Language and Speech Modalities

### Motivation

[DeepSeek-V3](https://arxiv.org/abs/2412.19437) introduced Multi-Token Prediction (MTP) — a training paradigm where the model predicts multiple future tokens simultaneously using shared transformer trunk with independent output heads. This approach has been shown to improve both training efficiency and inference throughput via speculative decoding.

More recently, [StepAudio 2.5](https://arxiv.org/abs/2605.23463) (StepFun, May 2026) extended MTP to the speech domain with **MTP-5**, demonstrating that a depth-5 MTP module can accelerate ASR inference to ~500 tokens/s while reducing inference cost by 80%. This suggests MTP is a modality-agnostic technique with broad applicability.

### Requested Features

1. **MTP Training Support for Language Models** — Implement the MTP training objective described in DeepSeek-V3 Sec 3.4, where K auxiliary prediction heads (sharing the main trunk) predict the next K tokens in parallel. The loss is computed as the weighted sum of the main cross-entropy loss and the MTP auxiliary losses.

2. **MTP-5 Training for Speech/Audio Models** — Extend MTP training to speech recognition/understanding tasks, following the StepAudio 2.5 recipe: an audio encoder → linear adapter → LLM backbone → MTP-5 module pipeline. This includes designing appropriate speech tokenization and alignment strategies so that the MTP heads can predict multiple future speech tokens per step.

3. **Configurable MTP Depth and Architecture** — Allow users to specify:
- Number of MTP layers (K, e.g., 1–5)
- Whether MTP heads share the transformer trunk (DeepSeek-V3 style) or use independent lightweight modules
- Loss weighting coefficient for auxiliary MTP losses

4. **Speculative Decoding Integration** — After MTP training, export the MTP module as a draft model for speculative decoding to improve inference throughput without quality degradation.

### References

- DeepSeek-AI. "DeepSeek-V3 Technical Report." arXiv:2412.19437, Dec 2024. https://arxiv.org/abs/2412.19437
- StepFun. "StepAudio 2.5 Technical Report." arXiv:2605.23463, May 2026. https://arxiv.org/abs/2605.23463

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.