NVIDIA / NVIDIA/TensorRT-LLM

[AutoDeploy][chore]: Better Sharding for Draft Models

Open
#13,174 0 comments 0 reactions 2 assignees View on GitHub

@greg-kwasniewski1 is already working on this.

Since May 5, 2026.

Speculative Decoding
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

Sharding for draft models is currently necessary, as we need consistency in attention sharding for draft and target to exist in a single KV cache manager. Even with multiple KV cache managers, sharding is a useful option for performance for the drafter.

Currently for sharding we first need to detect layers, which we do by understanding the embedding width and collecting operations between linear layers with opening/closing dimensions matching the embedding width. However, drafters for Eagle and MTP spec dec do not necessarily match these assumptions - they often have 2*embedding width for their input since they take hidden state AND tokens. This throws off the existing pattern matching used for detecting layers.

We have put in some code to automatically detect these divergences hardcoded for the specific Eagle / MTP models we currently support (Llama, Nemotron). However, this is very brittle, and likely to break with a slight divergence from a new Eagle model. We should brainstorm better ways to do this, or possibly replace altogether with new sharding infra.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.