NVIDIA / NVIDIA/TensorRT-LLM

[Feature]: AutoDeploy: DFlash speculative decoding support

Open
#14,843 0 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

AutoDeploy feature request Speculative Decoding
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

🚀 The feature, motivation and pitch

DFlash is a target-conditioned speculative decoding method that predicts a block of draft tokens in one parallel masked-token draft pass, using a persistent drafter-side context cache derived from accepted target hidden states. Rather than replaying accepted tokens through the draft transformer (as Eagle/MTP do), DFlash projects accepted target hidden states directly into the drafter and drafts multiple tokens at once.

DFlash is already supported in the PyTorch backend (configurable via DFlashDecodingConfig), but it is not available in AutoDeploy, which today supports Eagle3 and MTP one-model speculative decoding. This issue requests bringing DFlash to AutoDeploy as a sibling of the existing Eagle/MTP one-model path, so AutoDeploy users get the same draft-token speedups through AutoDeploy's export/compile/CUDA-graph pipeline.

Scope: The initial version implements the DFlash drafter algorithm itself and does not include tensor-parallel / sharding support, so it runs single-GPU (world_size == 1). Multi-GPU sharding is a natural follow-up. Accuracy and acceptance length are validated on GSM8K with meta-llama/Meta-Llama-3.1-8B-Instruct + z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat.

Alternatives

Users who need DFlash today must use the PyTorch backend. Keeping AutoDeploy and the PyTorch backend at parity on speculative-decoding methods is the motivation for this request.

Additional context

  • Existing PyTorch backend support: DFlashDecodingConfig in tensorrt_llm/llmapi/llm_args.py; implementation in tensorrt_llm/_torch/speculative/dflash.py.
  • Closest AutoDeploy precedent: the Eagle3/MTP one-model speculative-decoding path.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.