[feature] Offload KV cache from GPU (VRAM) to RAM
Open
- Dominant language
- C
- Stars
- 22.3k
- Forks
- 2.1k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
Running on RTX 6000 (96GB), 92GB are consumed with model loaded and 1 user (max 100K ctx). Remaining 4GB would fill up quickly with multiple users (not concurrent).
Contributor guide
Assessment
This issue has not been assessed yet.