Loading vicuna-13b-1.5 with single GPU in 16GB VM triggers system OOM
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
It seems that due to the way load_model() handles cuda devices, when running a worker with --num-gpus 1nodevice_map`` is set, which causes my VM to OOM:
[23502.263474] python3 invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[23502.263485] CPU: 5 PID: 1215 Comm: python3 Tainted: P OE 5.15.0-82-generic #91-Ubuntu
while loading shards:
Loading checkpoint shards: 33%|█████████████████████████████████ | 1/3 [00:14<00:28, 14.31s/it]
Killed
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the load_model() path used when a worker is started with --num-gpus 1, focusing on how CUDA devices and device_map are selected. Reproduce the single-GPU load with vicuna-13b-1.5 in a 16GB VM and verify that checkpoint loading completes without a system OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100