[Feature] Support Medusa decode (Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads)
Open
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Motivation
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads (https://github.com/FasterDecoding/Medusa) The proposed method can greatly improve the inference speed
### Related resources
https://github.com/FasterDecoding/Medusa
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.