vllm-project / vllm-project/aibrix
Prefix cache index not invalidated when a vLLM engine clears its prefix cache (reset/OOM/sleep)
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### Describe the bug
When an engine clears its prefix cache — `/reset_prefix_cache`, OOM-driven reset, or `/sleep level>=1` — it wipes the cache and enqueues `AllBlocksCleared`. We don't act on it: `handleAllBlocksCleared` is a no-op (`pkg/kvevent/handler.go`), and the only caller of `RemovePrefix` is pod unsubscribe, which doesn't fire here (the pod stays `Running`, `/health` still returns 200). So stale entries linger, and because the prefix router prefers the pod with the most matching blocks, it actively routes prefix traffic to the pod whose cache was just wiped — a cold miss.
The event alone isn't a reliable fix: vLLM publishes queued KV events only from inside a scheduler step (`update_from_output`), so an idle/sleeping engine may never flush `AllBlocksCleared`, and ZMQ delivery is lossy.
### Steps to Reproduce
1. Two replicas with the kv-event prefix router; warm a shared prefix on pod A.
2. `POST /reset_prefix_cache` on pod A (or `/sleep?level=1`).
3. Send the same prefix again → still routed to A, now a cold miss.
### Expected behavior
- `handleAllBlocksCleared` calls `RemovePrefix(model, loraID, podKey)` (already on the `SyncIndexer` interface).
- A reconcile against the scraped `vllm:engine_sleep_state` metric backstops dropped events (purge entries for pods reporting non-awake).
- When AIBrix drives sleep itself (separate sleep-mode RFC), it should also purge synchronously on `/sleep` — tracked there.
### Environment
AIBrix: main · vLLM: main (kv-events enabled).
Contributor guide
Research direction
Start in pkg/kvevent/handler.go, where handleAllBlocksCleared is currently a no-op, and trace the SyncIndexer interface and its RemovePrefix implementation. Review the /reset_prefix_cache and /sleep behavior plus the scraped vllm:engine_sleep_state metric, then verify that cleared or non-awake pods are removed from prefix routing while dropped events are covered by reconciliation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100