MoE disk offload
Đang mở
- Ngôn ngữ chính
- C
- Star
- 22.4k
- Fork
- 2.1k
- Merge trung bình
- 2 ngày 13 giờ
- Pull request đã merge (30 ngày)
- 5
Mô tả
Hi! I've been working on MoE disk offload for llama.cpp and got some promising [results](https://github.com/ggml-org/llama.cpp/discussions/23324). Managed to run a model nearly 2x larger than available RAM at ~9 tok/s.
The approach is pretty simple: keep N expert slots in memory and page missing ones from GGUF via pread with LRU eviction. The natural integration point in ds4 would be `tensor_expert_bytes()`, instead of returning a raw mmap pointer, check an LRU slot pool first and pread on miss. Don't have a 96GB machine to test myself, but thought you might be interested.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.