`pinned: true` model is freed after every request — idle-unload exclusion (#9305) ineffective on v4.7.1
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 49.2k
- Forks
- 4.5k
- Avg merge
- 1d 3m
- Merged PRs (30d)
- 239
Description
Follow-up to #9305 / #9309 (exclude models from idle unloading via pinned: true). On v4.7.1 a pinned: true model is not kept resident — its backend process is torn down after every request, so each call pays a full cold reload.
Environment
- LocalAI
v4.7.1, imagelocalai/localai:v4.7.1-gpu-nvidia-cuda-13, single NVIDIA 16 GB GPU,llama-cpp(cuda13) backend. - Model YAML:
backend: llama-cpp,pinned: true,context_size: 24576. - Runtime settings:
watchdog_enabled: false,single_backend: false,max_active_backends: 0(docs: "0 = unlimited"),memory_reclaimer_enabled: false,size_aware_eviction: false,force_eviction_when_busy: false.
Expected
Per the docs ("a pinned model is never evicted") and #9305: with no watchdog, no LRU limit, and pinned: true, the model should stay loaded across requests.
Actual
The backend is freed immediately after each response completes (not after any idle timeout):
INFO ... Predict request completed successfully
DEBUG Deleting process model="qwen3.5-9b" (pkg/model/process.go:49)
DEBUG Calling Free() to release GPU resources model="qwen3.5-9b"
INFO Backend process stopped id="qwen3.5-9b" exitCode="0"
This repeats for every single request across the log; the next request cold-loads (~10 s). Confirmed via nvidia-smi: the backend process disappears seconds after each response and reloads on the next call.
Repro
- Load a
pinned: truellama-cpp model on v4.7.1 with the runtime settings above. - Send two chat completions about a minute apart.
- Observe (via
nvidia-smi) the backend process disappear after the first completes and reload for the second.
Question
Is pinned: true expected to keep a model resident, or only to exempt it from eviction while some other path still frees it? If max_active_backends: 0 (documented as "unlimited") is implicated, the "0 = unlimited" behaviour and the #9305 exclusion don't appear to hold. Net effect: pinned currently provides no persistence benefit — every request pays a full cold model load.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/model/process.go at the deletion and Free() path shown in the logs, then trace how pinned and max_active_backends settings reach backend cleanup. Reproduce with the listed v4.7.1 runtime settings and two requests; done means a pinned model's backend process remains resident between requests without an idle timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100