mudler / mudler/LocalAI

`pinned: true` model is freed after every request — idle-unload exclusion (#9305) ineffective on v4.7.1

Open
#11,101 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

Follow-up to #9305 / #9309 (exclude models from idle unloading via pinned: true). On v4.7.1 a pinned: true model is not kept resident — its backend process is torn down after every request, so each call pays a full cold reload.

Environment
  • LocalAI v4.7.1, image localai/localai:v4.7.1-gpu-nvidia-cuda-13, single NVIDIA 16 GB GPU, llama-cpp (cuda13) backend.
  • Model YAML: backend: llama-cpp, pinned: true, context_size: 24576.
  • Runtime settings: watchdog_enabled: false, single_backend: false, max_active_backends: 0 (docs: "0 = unlimited"), memory_reclaimer_enabled: false, size_aware_eviction: false, force_eviction_when_busy: false.
Expected

Per the docs ("a pinned model is never evicted") and #9305: with no watchdog, no LRU limit, and pinned: true, the model should stay loaded across requests.

Actual

The backend is freed immediately after each response completes (not after any idle timeout):

INFO  ... Predict request completed successfully
DEBUG Deleting process model="qwen3.5-9b"           (pkg/model/process.go:49)
DEBUG Calling Free() to release GPU resources model="qwen3.5-9b"
INFO  Backend process stopped id="qwen3.5-9b" exitCode="0"

This repeats for every single request across the log; the next request cold-loads (~10 s). Confirmed via nvidia-smi: the backend process disappears seconds after each response and reloads on the next call.

Repro
  1. Load a pinned: true llama-cpp model on v4.7.1 with the runtime settings above.
  2. Send two chat completions about a minute apart.
  3. Observe (via nvidia-smi) the backend process disappear after the first completes and reload for the second.
Question

Is pinned: true expected to keep a model resident, or only to exempt it from eviction while some other path still frees it? If max_active_backends: 0 (documented as "unlimited") is implicated, the "0 = unlimited" behaviour and the #9305 exclusion don't appear to hold. Net effect: pinned currently provides no persistence benefit — every request pays a full cold model load.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/model/process.go at the deletion and Free() path shown in the logs, then trace how pinned and max_active_backends settings reach backend cleanup. Reproduce with the listed v4.7.1 runtime settings and two requests; done means a pinned model's backend process remains resident between requests without an idle timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.