ValueError: Size mismatch: memory_obj nbytes=3942656, gpu_buffer nbytes=15495168
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.9k
- Forks
- 1.9k
- Avg merge
- 4d 4h
- Merged PRs (30d)
- 141
Description
### version + env
**lmcache-0.5.1
vllm-0.24.0
GPU NVIDIA H100**
### Describe the bug
Use the following command to run vllm
`vllm serve /model/DeepSeek-V4-Flash --port 8001 --trust-remote-code --tensor-parallel-size 4 --enable-expert-parallel --max-model-len 1048576 --kv-cache-dtype fp8 --block-size 256 --speculative_config '{"method":"mtp","num_speculative_tokens":1}' --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --gpu-memory-utilization 0.85 --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":6555}}'`
Use the following command to run lmcache server
`lmcache server --l1-size-gb 512 --eviction-policy LRU --l1-init-size-gb 20 --chunk-size 1024 --port 6555 --max-gpu-workers 1 --l1-align-bytes 16384 --eviction-trigger-watermark 0.85 --eviction-ratio 0.10 --max-cpu-workers 8 --http-port 8002 --supported-transfer-mode lmcache_driven --l2-adapter '{"type": "fs_native", "base_path": "/model/lmcache/L2", "num_workers": 8, "max_capacity_gb": 10240,"use_odirect":false,"eviction":{"eviction_policy":"LRU","trigger_watermark":0.8,"eviction_ratio":0.2},"persist_enabled":true}' --coordinator-url http://10.240.243.203:31525 --coordinator-l2-event-reporting --coordinator-l2-event-flush-interval 1.0 --coordinator-heartbeat-interval 5.0 `
/model/lmcache/L2 is a gpfs share directory, The model is running normally, but the lmcache server reports an error as follows:
`
staging = build_staging_copies(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/gpu_connector/gpu_ops.py", line 131, in build_staging_copies
raise ValueError(
ValueError: Size mismatch: memory_obj nbytes=3942656, gpu_buffer nbytes=15495168
[2026-07-20 10:15:42,826] LMCache ERROR: Exception occurred while processing read prefetched results (storage_manager.py:347:lmcache.v1.distributed.storage_manager)
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/distributed/storage_manager.py", line 344, in read_prefetched_results
yield good_objs if all_good else None
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 1240, in retrieve
transfer_kv_per_object_group(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 449, in transfer_kv_per_object_group
_run_object_group_transfer_plan(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 364, in _run_object_group_transfer_plan
staging = build_staging_copies(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/gpu_connector/gpu_ops.py", line 131, in build_staging_copies
raise ValueError(
ValueError: Size mismatch: memory_obj nbytes=3942656, gpu_buffer nbytes=15495168
[2026-07-20 10:15:42,827] LMCache ERROR: Cannot retrieve keys due to exception (lmcache_driven_transfer.py:1254:lmcache.v1.multiprocess.modules.lmcache_driven_transfer)
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 1240, in retrieve
transfer_kv_per_object_group(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 449, in transfer_kv_per_object_group
_run_object_group_transfer_plan(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 364, in _run_object_group_transfer_plan
staging = build_staging_copies(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/gpu_connector/gpu_ops.py", line 131, in build_staging_copies
raise ValueError(
ValueError: Size mismatch: memory_obj nbytes=3942656, gpu_buffer nbytes=15495168
[2026-07-20 10:15:42,832] LMCache ERROR: Exception occurred while processing read prefetched results (storage_manager.py:347:lmcache.v1.distributed.storage_manager)
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/distributed/storage_manager.py", line 344, in read_prefetched_results
yield good_objs if all_good else None
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 1240, in retrieve
transfer_kv_per_object_group(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 449, in transfer_kv_per_object_group
_run_object_group_transfer_plan(
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py", line 364, in _run_object_group_transfer_plan
staging = build_staging_copies(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/lmcache/v1/gpu_connector/gpu_ops.py", line 131, in build_staging_copies
raise ValueError(
`
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lmcache/v1/gpu_connector/gpu_ops.py at build_staging_copies, then trace its callers in lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py and lmcache/v1/distributed/storage_manager.py. Reproduce the LMCache/vLLM setup and determine why the memory_obj and gpu_buffer sizes differ; done means prefetched results transfer without the reported ValueError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ai-infra-agents, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100