vllm-project / vllm-project/aibrix
KV event decoder drops group_idx/medium/lora_name, breaking prefix matching for hybrid and multi-tier models
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### Describe the bug
Our ZMQ decoder reads only the first four positional fields of vLLM's `BlockStored` (block_hashes, parent_block_hash, token_ids, block_size) and drops the rest. Upstream `BlockStored` (`vllm/distributed/kv_events.py`) now also carries `medium`, `lora_name`, `extra_keys`, and `group_idx`. Two of these affect correctness:
- `group_idx`: hybrid-attention models (sliding-window + full layers — Gemma, Ministral, Llama-4) hash the same token range into different KV-cache groups. Collapsing all groups into one hash space can produce false prefix matches.
- `medium`: with KV offloading, blocks live on CPU/remote tiers, but we count every block as a GPU hit, so prefix routing over-credits pods that only hold the prefix on a slower tier.
`lora_name` is also the canonical adapter id (`lora_id` is deprecated upstream); `prefix_cache.go` currently passes `loraID=-1` everywhere, so adapters aren't isolated in the index.
Relevant: `pkg/cache/kvcache/msgpack_decoder.go` (`parseEventArray`), `pkg/plugins/gateway/algorithms/prefix_cache.go`.
### Steps to Reproduce
1. Serve a hybrid-attention model (e.g. `google/gemma-2-9b-it`) with `--kv-events-config` enabled.
2. Send shared-prefix traffic across ≥2 replicas with the kv-event prefix router.
3. Observe requests routed to pods that don't actually have a GPU prefix hit.
### Expected behavior
Decoder parses `group_idx`/`medium`/`lora_name`; the index keys prefix entries by `(model, lora_name, group_idx)` and scores non-GPU mediums lower.
### Environment
AIBrix: main · vLLM: main (kv-events enabled).
Contributor guide
Assessment
This issue has not been assessed yet.