AlexsJones / AlexsJones/llmfit
MacBook Neo shows no models under "Good" or "Runnable" due to conservative KV cache filtering
- Langage dominant
- Rust
- Étoiles
- 36.5k
- Forks
- 2.3k
- Merge moyen
- 3 j 1 h
- PR mergées (30 j)
- 95
Description
### Describe the Bug
On an 8GB Apple Silicon device (specifically the new MacBook Neo with a binned A18 Pro chip), the TUI correctly detects system memory (displays `5.9 GB Available / 8.0 GB total`), but the model list completely zeroes out when the "Fit" filter is set to `Good` or `Runnable`.
Every single micro-model in the catalog (even 1.5B and 2B models like Qwen 3.5 and Gemma 3) is being pushed into the `Marginal` category. This seems to be caused by the scoring engine calculating memory requirements based on the model's maximum native context window (e.g., 32k tokens), which bloats the estimated KV cache footprint beyond the 5.9 GB threshold, even though these models run perfectly fine with smaller context sizes.
### Steps to Reproduce
1. Launch `llmfit` on an 8GB MacBook Neo.
2. Observe the top header line showing correct memory detection (~5.9 GB available).
3. Press `f` to cycle the Fit filter to `Runnable` or `Good`.
4. Observe that the TUI displays "No Models" matching the criteria.
5. Cycle the filter to `Marginal` or `All`, and notice that the ultra-compact models finally populate but with heavily penalized scores.
### Expected Behavior
Ultra-compact models (under 3B parameters with low quantization) should be categorized as `Runnable` or `Good` on an 8GB machine, perhaps with a warning or visual indicator that context length should be restricted, rather than hiding them from the runnable view entirely.
### Environment Details
- **Hardware:** MacBook Neo (A18 Pro, 5-core GPU variant)
- **RAM:** 8.0 GB total (5.9 GB reported available by `llmfit`)
- **OS:** macOS Tahoe 26.5.1
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.