[Feature] New Speculative Decoding Framework
Open
enhancement
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 275
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 33
Description
Due to the high CPU overhead in the existing speculative decoding framework, we are developing a brand new framework that significantly reduces CPU consumption and minimizes device-to-host synchronization.
### Worklist
- [x] basic feature [#305]
- support 1 & multi steps mtp
- support py model
- support PD-seperation
- support DP
- fast & async mtp process
- [x] vocab prune
- [x] support py model cuda graph [#516]
- [x] rm old framework
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.