AlexsJones / AlexsJones/llmfit
[Bug]: GPU fit grades against total VRAM while CPU paths grade against available RAM, so plan reports a fit on a full card
- 主要语言
- Rust
- 星标
- 36.3k
- 派生
- 2.3k
- 平均合并
- 2 天 18 小时
- 30 天内合并 PR
- 85
描述
### Bug description
`plan`'s fit grading compares against a different kind of memory figure on the GPU path than on the CPU paths, and only the CPU side reflects what is actually free.
After #813, `build_path_estimate` grades each path against the pool it uses:
```rust
// GPU — total VRAM
let available_vram = system.total_gpu_vram_gb.or(system.gpu_vram_gb).unwrap_or(0.0);
let fit = fit_level_for(path, min_vram, available_vram, rec_vram);
// CPU paths — *available* RAM
let fit = fit_level_for(path, min_ram, system.available_ram_gb, rec_ram);
```
`evaluate_current` does the same. So the CPU verdicts move as system load changes while the GPU verdict is blind to anything already resident on the card. You flagged the flappiness half of this on #813; this is the other half.
The cause looks like a capability gap rather than a deliberate split. `SystemSpecs::gpu_available_gb` exists but is Metal-only — `detect_gpu_available_gb()` calls `recommendedMaxWorkingSetSize()` and returns `None` on every non-macOS target, and it is only consulted at all when `unified_memory` is true. On CUDA/ROCm there is simply no free-VRAM figure to grade against: `try_nvidia_smi_*` queries `memory.total` and `addressing_mode`, never `memory.free`.
### Expected behavior
Either both pools describe total capacity ("could this machine ever run it") or both describe what is free right now ("will it fit at the moment") — and on a machine whose VRAM is largely occupied, the GPU path should not report that a model fits.
### Actual behavior
On a 24 GB RTX 3090 with a vLLM engine already holding ~21.8 GB, so **1.09 GB actually free**:
```
$ nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader
24576 MiB, 23010 MiB, 1112 MiB
$ llmfit --json plan "Qwen/Qwen3.6-35B-A3B" --context 32768 --quant Q4_K_M
needs VRAM : 23.85 GB
gpu fit_level : Marginal
current verdict : {'fit_level': 'Marginal', 'run_mode': 'Gpu', 'estimated_tps': 102.9}
```
`Marginal` reads as "tight but it fits", and `current` picks the GPU path at ~103 tok/s. Loading it would OOM immediately — it needs 23.85 GB against 1.09 GB free. Meanwhile a CPU-path verdict on the same machine *would* have accounted for the memory in use, because those arms use `available_ram_gb`.
This is not a regression from #813 — before it, the GPU arm compared the requirement against itself and was even less informative. It is the next layer down.
### Steps to reproduce
1. Occupy most of VRAM (any long-lived process: `vllm serve`, a llama.cpp server, a training job).
2. `llmfit --json plan "" --context 32768 --quant Q4_K_M`
3. The GPU path grades on total VRAM, so it reports `Marginal`/`Perfect` regardless of what is free.
### Diagnostic report (`llmfit doctor`)
```
# llmfit doctor report
Paste this whole report into a GitHub issue at https://github.com/AlexsJones/llmfit/issues — the raw tool output below is what lets detection bugs become regression tests. It contains hardware model names and driver info only.
- llmfit version: 1.1.6
- OS: linux (x86_64)
## Detected by llmfit
```
SystemSpecs {
total_ram_gb: 31.265457153320313,
available_ram_gb: 25.987503051757813,
total_cpu_cores: 24,
cpu_name: "AMD Ryzen 9 3900X 12-Core Processor",
has_gpu: true,
gpu_vram_gb: Some(
24.0,
),
total_gpu_vram_gb: Some(
24.0,
),
gpu_available_gb: None,
gpu_name: Some(
"NVIDIA GeForce RTX 3090",
),
gpu_count: 1,
unified_memory: false,
backend: Cuda,
gpus: [
GpuInfo {
name: "NVIDIA GeForce RTX 3090",
vram_gb: Some(
24.0,
),
backend: Cuda,
count: 1,
unified_memory: false,
},
],
cluster_mode: false,
cluster_node_count: 0,
}
```
## nvidia-smi (extended query)
```
None, 24576, NVIDIA GeForce RTX 3090
```
## nvidia-smi (standard query)
```
24576, NVIDIA GeForce RTX 3090
```
## rocm-smi --showmeminfo vram
```
(not available: No such file or directory (os error 2))
```
## rocm-smi --showproductname
```
(not available: No such file or directory (os error 2))
```
## sysfs DRM cards
```
card1: vendor=0x10de device=0x2204 driver=nvidia mem_info_vram_total=-
```
## lspci (display controllers)
```
2d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
```
## vulkaninfo --summary
```
(not available: No such file or directory (os error 2))
```
## npu-smi info
```
(not available: No such file or directory (os error 2))
```
## Provider app installs
```
LM Studio installed: false
Docker Desktop installed: false
ollama on PATH: false
```
```
Note `gpu_available_gb: None` — expected on CUDA, and precisely the gap.
### llmfit version
`llmfit 1.1.8`
### Operating system
Linux (Ubuntu 22.04.5, kernel 6.8.0)
### Affected component
Model fit analysis (scoring, fit levels)
### Mode used
Subcommand (`plan`) — `fit`/TUI share `SystemSpecs`, so the same asymmetry applies wherever VRAM headroom is graded.
### GPU details
```
NVIDIA GeForce RTX 3090, 24576 MiB total, driver 580.173.02
AMD Ryzen 9 3900X (24 threads), 32 GB RAM (31.27 GiB visible to the OS)
```
### Additional context
Three directions, and the choice is a semantics call like #806 turned out to be:
**A — both use total.** Switch the CPU arms to `total_ram_gb`. Smallest diff, and it makes verdicts reproducible between runs, which fixes the flappiness you raised. But it answers a weaker question, and the failure above stays: llmfit still says a model fits when the card is full.
**B — both use available.** Populate `gpu_available_gb` on discrete GPUs (`nvidia-smi --query-gpu=memory.free`, `rocm-smi --showmeminfo vram` gives used/total) and grade against it, keeping total as the fallback when the query fails. Most accurate, and it makes `plan` trustworthy on a busy machine — at the cost of the flappiness being real on both pools now, plus new detection code to maintain per backend.
**C — grade on total, report the shortfall.** Keep grading stable, but surface free memory when it is materially below total: a note on the path, or a `memory_available_now_gb` field. Consumers keep reproducible fit levels and gain the information needed to catch the OOM case.
My instinct is **B** for correctness with total as fallback, since "will this run right now" is what `plan` is asked, and `nvidia-smi` already exposes the number. But **C** is the conservative option and preserves the reproducibility argument, so I would rather have your call than guess as I did on #806.
Happy to implement whichever you prefer.
贡献指南
评估
这个 Issue 还没有评估数据。