vllm-project / vllm-project/aibrix

KV event decoder drops group_idx/medium/lora_name, breaking prefix matching for hybrid and multi-tier models

Open
#2,285 2 comments 0 reactions 0 assignees View on GitHub
area/gateway kind/bug
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### Describe the bug

Our ZMQ decoder reads only the first four positional fields of vLLM's `BlockStored` (block_hashes, parent_block_hash, token_ids, block_size) and drops the rest. Upstream `BlockStored` (`vllm/distributed/kv_events.py`) now also carries `medium`, `lora_name`, `extra_keys`, and `group_idx`. Two of these affect correctness:

- `group_idx`: hybrid-attention models (sliding-window + full layers — Gemma, Ministral, Llama-4) hash the same token range into different KV-cache groups. Collapsing all groups into one hash space can produce false prefix matches.
- `medium`: with KV offloading, blocks live on CPU/remote tiers, but we count every block as a GPU hit, so prefix routing over-credits pods that only hold the prefix on a slower tier.

`lora_name` is also the canonical adapter id (`lora_id` is deprecated upstream); `prefix_cache.go` currently passes `loraID=-1` everywhere, so adapters aren't isolated in the index.

Relevant: `pkg/cache/kvcache/msgpack_decoder.go` (`parseEventArray`), `pkg/plugins/gateway/algorithms/prefix_cache.go`.

### Steps to Reproduce

1. Serve a hybrid-attention model (e.g. `google/gemma-2-9b-it`) with `--kv-events-config` enabled.
2. Send shared-prefix traffic across ≥2 replicas with the kv-event prefix router.
3. Observe requests routed to pods that don't actually have a GPU prefix hit.

### Expected behavior

Decoder parses `group_idx`/`medium`/`lora_name`; the index keys prefix entries by `(model, lora_name, group_idx)` and scores non-GPU mediums lower.

### Environment

AIBrix: main · vLLM: main (kv-events enabled).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.