Local inference ignores system RAM and can't place MoE expert tensors separately
- 主要語言
- Rust
- 星號
- 54.2k
- 分支
- 6.2k
- 平均合併
- 3 天 2 小時
- 30 天內合併 PR
- 262
描述
**What problem would this solve?**
I have an RTX 5070 (12 GB) and 32 GB of RAM. Goose recommends me gemma-4-E4B. That's a very small model for this machine, and I think two things cause it.
First, the only placement control is `n_gpu_layers`, which moves whole layers. Attention and the feed-forward tensors in a layer go to the same place. For MoE models that's the wrong unit. The expert tensors are most of the file size but only a few of them run per token, while attention is small and runs every token. What you want is attention on the GPU for every layer and the experts in RAM. You can't say that with a layer count. Setting it to 16 pushes half the attention to CPU too, which is the part you wanted on the GPU.
Second, the memory check returns either GPU memory or system memory, never both added. It only looks at system RAM when there's no accelerator at all. That number decides which quantization gets recommended and what context size is picked automatically. So my 32 GB is invisible, and the recommendation comes from free VRAM alone.
Side effect of that: the CPU-only build recommends a bigger model than the CUDA build on the same machine, since only the CPU-only build is allowed to count RAM. Installing the GPU build gives Goose better hardware and a smaller budget.
This isn't NVIDIA-specific. The check uses ggml's device list and handles all accelerators in one branch, so Vulkan and Metal builds do the same thing.
Practical effect: the bigger MoE models in Goose's own featured list aren't really usable. A 26B MoE with ~4B active per token should run fine on 12 GB + 32 GB with the experts in RAM.
**What would a good outcome look like?**
Goose sizes and places the model against memory it can actually use, instead of one device's memory.
- Recommendation reflects the whole machine, and doesn't get smaller when I install the GPU build
- Auto context size uses the same number
- A MoE model that fits in VRAM + RAM runs with attention on the GPU, without me guessing at a layer count
- Still a sane default for people who don't want to touch it, still an override for people who do
Testable: same machine, same MoE model, compare recommended quant, context size and tok/s before and after. And check the CUDA build no longer recommends smaller than the CPU-only build.
**Possible approaches**
The `llama-cpp-2` version already in `Cargo.lock` has an automatic fitting call that works out layer count, tensor split and buffer-type overrides from whatever devices are present, and can pick context size in the same pass. It requires those params to still be at their defaults, which is where Goose already is when the user leaves the fields blank. That's the case that produces the bad recommendation, so it lines up. No new setting to maintain either.
Same crate has a dedicated helper for putting expert tensors on CPU. More direct for the MoE case and could sit behind a real setting, but it's more to maintain. Probably only worth it if the automatic path isn't good enough.
Some things I'm not sure about:
- Whether this should be automatic, manual, or automatic with an override, and what happens to `n_gpu_layers` if so
- The memory part might be worth fixing on its own. It's independent of placement and hits dense models the same way
- The fitting call is documented as not thread safe, it touches global logger state
- With more than one GPU the current check takes the largest card, not the total
- Free-memory numbers aren't the same thing across backends. Some report actual free device memory, some report the total, some report an API budget. On an integrated GPU it's system RAM anyway, so it isn't a second pool you can add. Probably an argument for letting the library decide rather than doing this arithmetic in Goose
**Additional context**
Where this lives:
- `crates/goose-local-inference/src/model.rs`, `n_gpu_layers` is the only placement field
- `crates/goose-local-inference/src/llamacpp/mod.rs`, model params at load time and the memory check
- `crates/goose-local-inference/src/hf_models.rs`, quant recommendation
- `crates/goose-local-inference/src/llamacpp/inference_engine.rs`, auto context estimate
Searching `crates/` and `ui/` for MoE-related placement terms only turns up quant labels parsed out of GGUF filenames and some chat template names. Nothing checks whether a model is MoE.
Not a duplicate of #10617, that one was about exposing GPU info to the agent over MCP so it could pick models for external runtimes. This is Goose's own local inference.
- [x] I have verified this does not duplicate an existing feature request
貢獻指南
評估
這個 Issue 還沒有評估資料。