Prompt cache: identical resend never hits, partial match discards everything, and nothing is sent during prefill

Summary

On the OpenAI-compatible server, a long agentic conversation can reach a state it
never recovers from: every request re-prefills the entire conversation, the client
times out because nothing is sent to it during prefill, the retry queues behind the
request still running, and the loop sustains itself.

Observed with OpenCode against qwen3.5:9b on a 146-message, ~98k-token
conversation. Four distinct contributing defects are below, all located in
PromptCache / AutoModel::_shared_insert / the streaming path on main
(92f3f13).

Environment

• OS: Ubuntu 26.04.1 LTS (kernel 7.0.0-31-generic)
• NPU driver: NPU FW 1.1.2.64, amdxdna 0.7
• flm version: FLM v1.0.6
• flm validate:

[Linux]  Kernel: 7.0.0-31-generic
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.7
[Linux]  Memlock Limit: infinity

Command that fails

flm serve qwen3.5:9b --ctx-len 262144

then drive it from an agentic client (OpenCode) until the conversation is large,
and let one request time out.

---

1. An identical resend can never hit the prompt cache

PromptCache::can_use_cache (src/include/prompt_cache.hpp:176-179):

const size_t prefix_len = messages.size() - 2;
const bool can_use_message =
    message_checksums_.size() <= prefix_len &&
    matched == message_checksums_.size();

message_checksums_ is updated to the full incoming list after every request,
so a resend of the same conversation has message_checksums_.size() == messages.size()
and fails <= messages.size() - 2. The - 2 demands two new messages; a retry has
none.

This is exactly what a client sends after a timeout, so the retry costs a full
re-prefill — slower than the request that timed out — and the next retry does the
same. Nothing breaks the cycle.

The same bound appears in can_use_message_cache at line 130.

2. A partial match discards the whole cache

With (1) worked around locally, the real conversation then produced:

Prompt cache miss: conversation diverged at message 145 [cached=146 matched=145 incoming=146 tools_matched=yes]
Clearing context... (146 messages will be re-prefilled)

145 of 146 messages match — only the last differs — and all 146 are re-prefilled.
can_use_cache requires matched == message_checksums_.size(), and
AutoModel::_shared_insert independently discards a partial token match:

if (skip_count != idx) { clear_context(); skip_count = 0; }

A single edited or regenerated trailing message therefore costs the entire
conversation.

3. restore() is only faithful when a prefill() follows it

Fixing (1) exposes the fully-cached case, where there is nothing left to prefill:

Use cached prompt!
Matched 146 out of 146 messages (0 new to prefill).
Restoring checkpoint at context length 98407

_chunked_insert then computes zero chunks and returns a default-constructed
(empty) logits buffer straight into sampler->sample().

Three ways around that were implemented and measured on qwen3.5:9b. The probe
buries a value in a 6071-token prompt and asks for it back; four cold runs return
it 4/4 with byte-identical wording.

approach	result
Hold back the last token, roll the kv cache back one position with set_context_length(), prefill that token	3/4 — one outright refusal, one XYZZY-7391 → XYYZY-7391
Truncate to the common prefix on a partial match	cold run answered Lima., truncated-cache run answered Peru is a country, not a city
Save the prefill logits beside the checkpoint and sample from them with no prefill at all	3/4 — XYZZY-7391 → XYYZV - 8245
Ordinary hit: restore() followed by a real prefill of the new tail	4/4 exact

So restore() itself is faithful, but only in combination with a subsequent
prefill(): the engine appears to re-establish state there that a bare restore does
not. Note set_context_length() does adjust the length — the token accounting and
timings looked right (6047 of 6048 reused, 15.7s → 2.0s) — but the output was wrong.

Because the engine is distributed as a prebuilt library, this cannot be addressed
from the open-source layer.

Ask: either a faithful zero-prefill resume after restore(), or a supported kv
truncation primitive so a partial prefix can be reused. (2) is worth far more than
(3) in practice — reusing 145 of 146 messages would make the pathological case
disappear.

4. Nothing is written to the socket during prefill

HttpSession::write_streaming_response sends the HTTP headers on its first call,
and in the streaming chat path that call cannot happen until the first token is
generated — insert() runs to completion first.

For a ~98k-token prompt that is minutes during which the client receives zero
bytes, not even response headers, and cannot distinguish a working server from a
dead one. Its only recourse is to time out and retry, which is what turns a slow
request into the loop above.

Measured on qwen3.5:9b with a 9038-token prompt: time to first byte equals the
whole prefill. Emitting an SSE comment (: keepalive, ignored by clients) before
prefill and periodically during it takes TTFB to 0.002s. is_cancelled() is
already invoked once per prefill chunk and can carry the heartbeat with no new
plumbing.

---

Two minor defects found along the way

size_t underflow in the cache-hit log (src/server/rest_handler.cpp:1167-1170):

size_t matched_rounds = cache_info.matched_rounds + (auto_chat_engine->check_using_checkpint() ? 0 : 1);
... std::to_string(cache_info.total_rounds - matched_rounds) ...

The + 1 can push matched_rounds past total_rounds, and the subtraction is
unsigned. Observed output:

Matched 5 out of 4 messages (18446744073709551615 new to prefill).

cache_match_info_t::tools_matched is never set on the early-return path
(src/include/prompt_cache.hpp:154). can_use_cache returns for
messages.size() <= 2 before assigning it, so it keeps its default false. Any
caller reporting a miss reason from it will blame the tool definitions for every new
conversation.

---

Reproduction

The probe used for (3), which distinguishes a faithful cache hit from a corrupting
one — bury a value in a large prompt, then ask for it back over several trials and
compare against cold runs:

filler = "The quick brown fox jumps over the lazy dog. " * 300
SECRET = "XYZZY-7391"
prompt = filler + f"\n\nRemember this: the secret code is {SECRET}.\n\n" + filler
conv = [{"role": "system",    "content": "You are terse. Answer in one short sentence."},
        {"role": "user",      "content": prompt + "\n\nWhat is the capital of France?"},
        {"role": "assistant", "content": "Paris."},
        {"role": "user",      "content": "What is the secret code stated above?"}]
# POST conv to /v1/chat/completions repeatedly; a cold run returns the value 4/4
# byte-identically, so any variation indicates the cache path is not faithful.

Happy to open PRs for (1), (4) and the two minor defects — those are all in the
open-source layer. (2) and (3) need a decision from someone who can see the engine.

Start with PromptCache in src/include/prompt_cache.hpp, AutoModel::_shared_insert, the streaming chat path, and src/server/rest_handler.cpp. Reproduce with flm serve qwen3.5:9b --ctx-len 262144 and the supplied Python probe, then trace cache hits, partial matches, restore behavior, and prefill streaming. Done requires a scoped decision for the engine-dependent cases plus verified fixes for the open-source defects.