intel / intel/llm-scaler

cpu offloading doesn't work

Open
#556 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

```
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/offloader/uva.py", line 56, in wrap_modules
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] modules = [self._maybe_offload_to_cpu(module) for module in modules_generator]
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/offloader/uva.py", line 104, in _maybe_offload_to_cpu
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] p.data = get_accelerator_view_from_cpu_tensor(cpu_data)
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] File "/opt/venv/lib/python3.12/site-packages/vllm/utils/torch_utils.py", line 798, in get_accelerator_view_from_cpu_tensor
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] assert cpu_tensor.is_pinned(), "CPU tensor must be pinned"
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=43) ERROR 07-22 16:03:46 [multiproc_executor.py:870] AssertionError: CPU tensor must be pinned
```

Contributor guide

Open the contributing guide

Research direction

Start with vllm/model_executor/offloader/uva.py at wrap_modules and _maybe_offload_to_cpu, then inspect vllm/utils/torch_utils.py at get_accelerator_view_from_cpu_tensor. Reproduce the reported CPU offloading failure and trace why the tensor is not pinned; done means the same offloading path no longer raises the assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.