[AutoDeploy][chore]: Better Sharding for Draft Models
@greg-kwasniewski1 is already working on this.
Since May 5, 2026.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Sharding for draft models is currently necessary, as we need consistency in attention sharding for draft and target to exist in a single KV cache manager. Even with multiple KV cache managers, sharding is a useful option for performance for the drafter.
Currently for sharding we first need to detect layers, which we do by understanding the embedding width and collecting operations between linear layers with opening/closing dimensions matching the embedding width. However, drafters for Eagle and MTP spec dec do not necessarily match these assumptions - they often have 2*embedding width for their input since they take hidden state AND tokens. This throws off the existing pattern matching used for detecting layers.
We have put in some code to automatically detect these divergences hardcoded for the specific Eagle / MTP models we currently support (Llama, Nemotron). However, this is very brittle, and likely to break with a slight divergence from a new Eagle model. We should brainstorm better ways to do this, or possibly replace altogether with new sharding infra.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.