antirez / antirez/ds4

MoE disk offload

Open
#202 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.4k
Forks
2.1k
Avg merge
2d 13h
Merged PRs (30d)
5

Description

Hi! I've been working on MoE disk offload for llama.cpp and got some promising [results](https://github.com/ggml-org/llama.cpp/discussions/23324). Managed to run a model nearly 2x larger than available RAM at ~9 tok/s.

The approach is pretty simple: keep N expert slots in memory and page missing ones from GGUF via pread with LRU eviction. The natural integration point in ds4 would be `tensor_expert_bytes()`, instead of returning a raw mmap pointer, check an LRU slot pool first and pread on miss. Don't have a 96GB machine to test myself, but thought you might be interested.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.