openclaw / openclaw/openclaw-windows-node
local-ai: llama-server automatically unloads kv-cache causing 20 second hit on TTFT
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 295
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 99
Description
After some inactivity, llama-server unloads the kv-cache (and possibly also the model). This causes a 20 second hit on time-to-first-token (TTFT) where 27k tokens of the system prompt are pre-filled.
Investigate:
- Does kv-cache have timeout based eviction and if yes, can we avoid evicting system prompt?
- Does the model weights also unload?
Change llama-cpp settings to correctly address these.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting llama-server's handling of kv-cache timeout eviction and whether model weights are unloaded after inactivity. Review the current llama-cpp settings and measure the effect on the 27k-token system prompt and TTFT. Done means the unload behavior is confirmed and the settings address the unwanted 20-second delay.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100