LMCache / LMCache/LMCache

[Bug] lmcache bench engine fails when vLLM uses --served-model-name: model alias vs HF path mismatch

Open
#3,809 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Python
Stars
11.9k
Forks
1.9k
Avg merge
4d 4h
Merged PRs (30d)
141

Description

Describe the bug

When vLLM is started with --served-model-name (to expose a model under an alias different from its HF path), lmcache bench engine cannot run because a single --model value is used for three incompatible purposes, and the LMCache server's /status endpoint does not expose the served-model-name alias.

To Reproduce

  1. Start vLLM with an alias:

vllm serve /llm2/Moonshotai/Kimi-K2.5
--served-model-name kimi-k2.5
--tensor-parallel-size 8
--kv-transfer-config '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both"}'
2. Start LMCache server:

lmcache server --l1-size-gb 1024 --eviction-policy LRU
3. Run bench with the alias:

lmcache bench engine
--engine-url http://localhost:8000
--workload long-doc-qa
--lmcache-url http://localhost:8080
--model kimi-k2.5
Error:

RuntimeError: Model 'kimi-k2.5' not found on LMCache server. Available: /llm2/Moonshotai/Kimi-K2.5
4. Run bench with the HF path (workaround attempt):

lmcache bench engine ... --model /llm2/Moonshotai/Kimi-K2.5
Error:

ErrorInfo(message='The model /llm2/Moonshotai/Kimi-K2.5 does not exist.', type='NotFoundError', param='model', code=404)
There is no --model value that satisfies both sides.

Root Cause

The bench tool's --model / config.model value is used for three mutually incompatible purposes:

Purpose Location Required value
Match LMCache /status config.py:_find_model_meta HF model path (vllm_config.model_config.model)
vLLM OpenAI API requests request_sender.py served-model-name alias
Tokenizer loading workloads/prefix_suffix_tuner.py HF model path
The LMCache server's cache_context_meta stores model_name from vllm_config.model_config.model (the HF path), while vLLM's /v1/models returns the --served-model-name alias. The MP connector (lmcache_mp_connector.py:532,542) passes vllm_config.model_config.model to the adapter, never served_model_name. So the /status response has no alias field to reconcile the two.

auto_detect_model() fetches from /v1/models and returns the alias — correct for API requests but wrong for LMCache matching and tokenizer loading.

Expected behavior

lmcache bench engine --model kimi-k2.5 (the alias) should work end-to-end when vLLM is started with --served-model-name kimi-k2.5.

Suggested fix

Server side: Propagate served_model_name through the registration chain so /status exposes it alongside model_name:
MP connectors pass vllm_config.model_config.served_model_name to adapters
REGISTER_KV_CACHE / CB_REGISTER_KV_CACHE protocols carry served_model_name (append as last payload, str | None)
ContextEntry / NonGPUContextEntry / BlendModule meta store it
report_status() includes "served_model_name" in each meta entry
Same for SGLang and TRT-LLM adapters
Bench side:
_find_model_meta matches against both model_name and served_model_name; falls back to model_name-only when served_model_name is absent (backward compat with old servers)
Tokenizer loading falls back to the HF path from /status when the alias fails to load

Environment

LMCache: v0.4.7
vLLM connector: LMCacheMPConnector
OS: Linux
Workaround
Remove --served-model-name from the vLLM launch command so the alias equals the HF path, then pass the HF path to --model.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with config.py:_find_model_meta, request_sender.py, workloads/prefix_suffix_tuner.py, and lmcache_mp_connector.py to trace how the model value is used. Then inspect the REGISTER_KV_CACHE/CB_REGISTER_KV_CACHE paths, ContextEntry metadata, and report_status(). Done means bench engine accepts the served-model-name alias while matching LMCache status and loading the tokenizer from the HF path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, cli, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.