[AutoDeploy][Feature]: Spec Dec + Guided Decoding Support
Open
Nobody has claimed this yet.
AutoDeploy
feature request
Speculative Decoding
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
🚀 The feature, motivation and pitch
Spec dec + guided decoding together cause some complications - roughly speaking, the sampling needs to do guided decoding, but then the draft tokens also need to be guided otherwise they will never be accepted.
Currently we just check that we don't have both enabled at the same time for AutoDeploy.
Alternatives
No response
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.