huggingface / huggingface/ratchet
Vectorized KV-Cache
Open
- Dominant language
- Rust
- Stars
- 771
- Forks
- 44
- Avg merge
- 40m
- Merged PRs (30d)
- 1
Description
Currently our cache operation is just scalar: https://github.com/huggingface/ratchet/blob/master/crates/ratchet-core/src/ops/cache.rs#L154
We need to vectorize it.
Contributor guide
Research direction
Start in crates/ratchet-core/src/ops/cache.rs around line 154 and trace the current scalar cache operation and its callers. Determine the vectorized KV-cache behavior needed and how the existing operation's behavior should be preserved. Done means the cache operation is vectorized rather than scalar.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100