LMCache / LMCache/LMCache

ValueError: Size mismatch: memory_obj nbytes=3942656, gpu_buffer nbytes=15495168

Open
#4,179 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
11.9k
Forks
1.9k
Avg merge
4d 4h
Merged PRs (30d)
141

Description

### version + env
**lmcache-0.5.1
vllm-0.24.0
GPU NVIDIA H100**

### Describe the bug
Use the following command to run vllm

`vllm serve /model/DeepSeek-V4-Flash --port 8001 --trust-remote-code --tensor-parallel-size 4 --enable-expert-parallel --max-model-len 1048576 --kv-cache-dtype fp8 --block-size 256 --speculative_config '{"method":"mtp","num_speculative_tokens":1}' --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --gpu-memory-utilization 0.85 --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":6555}}'`
Use the following command to run lmcache server

`lmcache server --l1-size-gb 512 --eviction-policy LRU --l1-init-size-gb 20 --chunk-size 1024 --port 6555 --max-gpu-workers 1 --l1-align-bytes 16384 --eviction-trigger-watermark 0.85 --eviction-ratio 0.10 --max-cpu-workers 8 --http-port 8002 --supported-transfer-mode lmcache_driven --l2-adapter '{"type": "fs_native", "base_path": "/model/lmcache/L2", "num_workers": 8, "max_capacity_gb": 10240,"use_odirect":false,"eviction":{"eviction_policy":"LRU","trigger_watermark":0.8,"eviction_ratio":0.2},"persist_enabled":true}' --coordinator-url http://10.240.243.203:31525 --coordinator-l2-event-reporting --coordinator-l2-event-flush-interval 1.0 --coordinator-heartbeat-interval 5.0 `

/model/lmcache/L2 is a gpfs share directory, The model is running normally, but the lmcache server reports an error as follows:

`
staging = build_staging_copies(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/gpu_connector/gpu_ops.py", line 131, in build_staging_copies
raise ValueError(
ValueError: Size mismatch: memory_obj nbytes=3942656, gpu_buffer nbytes=15495168
[2026-07-20 10:15:42,826] LMCache ERROR: Exception occurred while processing read prefetched results (storage_manager.py:347:lmcache.v1.distributed.storage_manager)
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/distributed/storage_manager.py", line 344, in read_prefetched_results
yield good_objs if all_good else None
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 1240, in retrieve
transfer_kv_per_object_group(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 449, in transfer_kv_per_object_group
_run_object_group_transfer_plan(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 364, in _run_object_group_transfer_plan
staging = build_staging_copies(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/gpu_connector/gpu_ops.py", line 131, in build_staging_copies
raise ValueError(
ValueError: Size mismatch: memory_obj nbytes=3942656, gpu_buffer nbytes=15495168
[2026-07-20 10:15:42,827] LMCache ERROR: Cannot retrieve keys due to exception (lmcache_driven_transfer.py:1254:lmcache.v1.multiprocess.modules.lmcache_driven_transfer)
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 1240, in retrieve
transfer_kv_per_object_group(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 449, in transfer_kv_per_object_group
_run_object_group_transfer_plan(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 364, in _run_object_group_transfer_plan
staging = build_staging_copies(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/gpu_connector/gpu_ops.py", line 131, in build_staging_copies
raise ValueError(
ValueError: Size mismatch: memory_obj nbytes=3942656, gpu_buffer nbytes=15495168
[2026-07-20 10:15:42,832] LMCache ERROR: Exception occurred while processing read prefetched results (storage_manager.py:347:lmcache.v1.distributed.storage_manager)
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/distributed/storage_manager.py", line 344, in read_prefetched_results
yield good_objs if all_good else None
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 1240, in retrieve
transfer_kv_per_object_group(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 449, in transfer_kv_per_object_group
_run_object_group_transfer_plan(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 364, in _run_object_group_transfer_plan
staging = build_staging_copies(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/gpu_connector/gpu_ops.py", line 131, in build_staging_copies
raise ValueError(
`

Image

Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lmcache/v1/gpu_connector/gpu_ops.py at build_staging_copies, then trace its callers in lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py and lmcache/v1/distributed/storage_manager.py. Reproduce the LMCache/vLLM setup and determine why the memory_obj and gpu_buffer sizes differ; done means prefetched results transfer without the reported ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ai-infra-agents, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.