[Feature]: AutoDeploy: DFlash speculative decoding support
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
🚀 The feature, motivation and pitch
DFlash is a target-conditioned speculative decoding method that predicts a block of draft tokens in one parallel masked-token draft pass, using a persistent drafter-side context cache derived from accepted target hidden states. Rather than replaying accepted tokens through the draft transformer (as Eagle/MTP do), DFlash projects accepted target hidden states directly into the drafter and drafts multiple tokens at once.
DFlash is already supported in the PyTorch backend (configurable via DFlashDecodingConfig), but it is not available in AutoDeploy, which today supports Eagle3 and MTP one-model speculative decoding. This issue requests bringing DFlash to AutoDeploy as a sibling of the existing Eagle/MTP one-model path, so AutoDeploy users get the same draft-token speedups through AutoDeploy's export/compile/CUDA-graph pipeline.
Scope: The initial version implements the DFlash drafter algorithm itself and does not include tensor-parallel / sharding support, so it runs single-GPU (world_size == 1). Multi-GPU sharding is a natural follow-up. Accuracy and acceptance length are validated on GSM8K with meta-llama/Meta-Llama-3.1-8B-Instruct + z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat.
Alternatives
Users who need DFlash today must use the PyTorch backend. Keeping AutoDeploy and the PyTorch backend at parity on speculative-decoding methods is the motivation for this request.
Additional context
- Existing PyTorch backend support:
DFlashDecodingConfigintensorrt_llm/llmapi/llm_args.py; implementation intensorrt_llm/_torch/speculative/dflash.py. - Closest AutoDeploy precedent: the Eagle3/MTP one-model speculative-decoding path.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.