Implement speculative decoding
Open
enhancement
- Dominant language
- Python
- Stars
- 18
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Speculative decoding is implemented in `LitGPT`.
**Describe the solution you'd like**
The `LitGPT` code just needs to be ported to use KV caches.
Contributor guide
Research direction
Start by reading the existing speculative decoding implementation in LitGPT and then inspect this repository's KV-cache entry points. Determine how the LitGPT behavior maps to KV caches; the work is done when speculative decoding is ported and functions through those caches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100