vllm-project / vllm-project/aibrix

Prefix cache index not invalidated when a vLLM engine clears its prefix cache (reset/OOM/sleep)

Open
#2,287 5 comments 0 reactions 0 assignees View on GitHub
area/gateway kind/bug
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### Describe the bug

When an engine clears its prefix cache — `/reset_prefix_cache`, OOM-driven reset, or `/sleep level>=1` — it wipes the cache and enqueues `AllBlocksCleared`. We don't act on it: `handleAllBlocksCleared` is a no-op (`pkg/kvevent/handler.go`), and the only caller of `RemovePrefix` is pod unsubscribe, which doesn't fire here (the pod stays `Running`, `/health` still returns 200). So stale entries linger, and because the prefix router prefers the pod with the most matching blocks, it actively routes prefix traffic to the pod whose cache was just wiped — a cold miss.

The event alone isn't a reliable fix: vLLM publishes queued KV events only from inside a scheduler step (`update_from_output`), so an idle/sleeping engine may never flush `AllBlocksCleared`, and ZMQ delivery is lossy.

### Steps to Reproduce

1. Two replicas with the kv-event prefix router; warm a shared prefix on pod A.
2. `POST /reset_prefix_cache` on pod A (or `/sleep?level=1`).
3. Send the same prefix again → still routed to A, now a cold miss.

### Expected behavior

- `handleAllBlocksCleared` calls `RemovePrefix(model, loraID, podKey)` (already on the `SyncIndexer` interface).
- A reconcile against the scraped `vllm:engine_sleep_state` metric backstops dropped events (purge entries for pods reporting non-awake).
- When AIBrix drives sleep itself (separate sleep-mode RFC), it should also purge synchronously on `/sleep` — tracked there.

### Environment

AIBrix: main · vLLM: main (kv-events enabled).

Contributor guide

Open the contributing guide

Research direction

Start in pkg/kvevent/handler.go, where handleAllBlocksCleared is currently a no-op, and trace the SyncIndexer interface and its RemovePrefix implementation. Review the /reset_prefix_cache and /sleep behavior plus the scraped vllm:engine_sleep_state metric, then verify that cleared or non-awake pods are removed from prefix routing while dropped events are covered by reconciliation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.