cpu offloading doesn't work
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
```
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/offloader/uva.py", line 56, in wrap_modules
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] modules = [self._maybe_offload_to_cpu(module) for module in modules_generator]
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/offloader/uva.py", line 104, in _maybe_offload_to_cpu
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] p.data = get_accelerator_view_from_cpu_tensor(cpu_data)
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] File "/opt/venv/lib/python3.12/site-packages/vllm/utils/torch_utils.py", line 798, in get_accelerator_view_from_cpu_tensor
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] assert cpu_tensor.is_pinned(), "CPU tensor must be pinned"
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] AssertionError: CPU tensor must be pinned
```
Contributor guide
Research direction
Start with vllm/model_executor/offloader/uva.py at wrap_modules and _maybe_offload_to_cpu, then inspect vllm/utils/torch_utils.py at get_accelerator_view_from_cpu_tensor. Reproduce the reported CPU offloading failure and trace why the tensor is not pinned; done means the same offloading path no longer raises the assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100