kvcache-ai / kvcache-ai/ktransformers

cache_salt/extra_key are computed & carried but never delivered to Req — radix-cache tenant isolation is a no-op (regression vs upstream sglang)

Open
#2,184 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

```text
ktransformers : 0.7.0.post1 (main @ 948129159aa750075a0264aca16dba061cc6e6ae, 2026-08-28)
sglang : sglang-kt (kvcache-ai/sglang @ 3bb39fca90ea8cd5d643866fdb6c8371ffc162de, "fix(glm5-next): accept MTP index sharing metadata")
pip install -e python/ (from fork source), transformers-kt 5.6.0.post3
model : Qwen/Qwen2.5-0.5B-Instruct (HF)
platform : Ubuntu 22.04, kernel 6.8.0-136-generic
python : 3.12.11 (conda env "sglang-kt"); torch 2.9.1+cu128; flashinfer 0.6.3
cpu : 2 × AMD EPYC 7763 64-Core Processor (256 threads)
gpu : NVIDIA GeForce RTX 3090 24GB ×8, driver 595.80 (CUDA 13.2), GPU 0 used
server : python -m sglang.launch_server --model-path /home/cjb/kt-models/qwen25-05b \
--host 0.0.0.0 --port 31000 --mem-fraction-static 0.6 --disable-cuda-graph \
--enable-metrics --log-level info (radix cache enabled by default, page_size=1, lru eviction)
```

### Reproduction

Request sequence (OpenAI `/v1/completions`, same prompt, **5 semantically distinct `cache_salt`/`extra_key` contexts**):

```text
T1: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"tenant-alpha"}
T2: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"tenant-beta"}
T3: {"prompt": P, "max_tokens":1, "temperature":0, "extra_key":"tenant-gamma"}
T4: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"a", "extra_key":"bc"}
T5: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"ab", "extra_key":"c"}
```
(P is a fixed ~450-token text; the full JSON of P and all 5 requests is in the attached scripts.)

Expected: different `(cache_salt, extra_key)` contexts → distinct radix namespaces → no cross-hits (T2–T5 should recompute prefill after T1's cold start).

Actual (scheduler INFO log, per-request `#cached-token`):

```text
T1 Prefill batch, #new-token: 457, #cached-token: 2 (cold start)
T2 Prefill batch, #new-token: 1, #cached-token: 458 (full hit on T1's KV!)
T3 Prefill batch, #new-token: 1, #cached-token: 458 (hit)
T4 Prefill batch, #new-token: 1, #cached-token: 458 (hit)
T5 Prefill batch, #new-token: 1, #cached-token: 458 (hit)
```

I.e. **all 5 semantically unrelated contexts share a single KV-cache namespace** — the `cache_salt`/`extra_key` isolation contract is a complete no-op at runtime.

Code-level break (fork @ 3bb39fca) — the feature is computed, carried layer by layer, then **dropped when `Req` is constructed**:

```text
serving_base.py:146-157 _compute_extra_key() merges cache_salt+extra_key (called by serving_chat:318 / serving_completions:119 / serving_responses:283)
io_struct.py:208 / :724 GenerateReqInput.extra_key → TokenizedGenerateReqInput.extra_key
tokenizer_manager.py:964 extra_key=obj.extra_key
managers/scheduler.py:1504-1536 handle_generate_request constructs Req(...) WITHOUT extra_key=recv_req.extra_key (no read of recv_req.extra_key anywhere in the repo)
schedule_policy.py:194-236 / schedule_batch.py:938 / mem_cache/radix_cache.py consumers intact, but always receive None
```

Upstream comparison (regression evidence): sgl-project/sglang main `managers/scheduler.py:2549` passes `extra_key=recv_req.extra_key` at **the same construction site** (also :2863; :2464-2467 additionally uses extra_key for the elastic-EP namespace). In the fork, the serving layer, io_struct, the `Req.__init__` parameter, and the radix-cache namespace implementation (radix_cache.py:357-364 docstring: "Isolate KV cache lines for different LoRA / adapter IDs … cache version, or retrieval augmentation context") are all retained — only the scheduler hand-off is missing.

### Others

**Impact scope** (no over-claiming):
- For plain-text models identical tokens ⇒ identical KV, so cross-context hits do not change outputs. Actual harm: the documented isolation contract is void (deployments that rely on salt semantics — custom logit processors, retrieval-augmented contexts, per-tenant state — get silently wrong outputs), plus a cross-tenant prefix-cache hit side channel (timing/metadata leakage).
- LoRA isolation remains effective (lora_id is passed at scheduler.py:1513 and folded into the namespace by `Req.__init__`) — statically inferred, not GPU-tested this time.

**Suggested fix**:
1. Main fix (1 line, restores upstream behavior): add `extra_key=recv_req.extra_key` to the `Req(...)` call in `scheduler.handle_generate_request` (`fix_extra_key_delivery.patch` attached; py_compile verified; `TokenizedGenerateReqInput.extra_key` field already exists, io_struct:724).
2. **Must be done together**: the separator-less concatenation in serving_base.py:146-157 (`"".join([cache_salt, extra_key])`) and schedule_batch.py:632-638 (`(extra_key or "") + lora_id`) permits semantic collisions — `(salt="a", ek="bc") ≡ (salt="ab", ek="c") ≡ (ek=)` — which become exploitable the moment fix #1 lands (reproduced against the fork's real RadixCache in `verify_cache_identity.py`: CROSS-CONTEXT HIT). Recommend delimiter/hash encoding (length-prefixed or sha256).

**Please confirm**: is this a regression or intentionally disabled? (Upstream passes the same field at the same site; the fork has no visible git history marking a deliberate removal, so we cannot rule intent out ourselves.)

Contributor guide

Open the contributing guide

Research direction

Start at managers/scheduler.py:1504-1536 and compare the Req construction with the upstream scheduler hand-off. Trace extra_key through serving_base.py, io_struct.py, tokenizer_manager.py, schedule_batch.py, and mem_cache/radix_cache.py. Run the attached verify_cache_identity.py and py_compile checks; done means distinct cache_salt/extra_key contexts do not share radix-cache entries, including separator-collision cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance, security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.