NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Excessive memory usage during custom model checkpoint loading & saving
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
to_hf and from_hf create additional tensors on device for the conversion which leads to a lot of overhead memory usage. This is particularly obvious in PEFT usecases. For example, if you try loading nemotron-v3-nano 30B onto a GPU with 80GB of VRAM, it should load however it OOMs at load.
Casting tensor creation to CPU might also not work universally in situations where there is unified memory. We need to find a way where we can load in place or one at a time.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the to_hf and from_hf conversion paths first, then reproduce the nemotron-v3-nano 30B PEFT loading case on an 80GB GPU while observing device allocations. Done means checkpoint conversion no longer creates avoidable additional device tensors and the stated load case avoids OOM; the issue does not name a test or source file, so those must be located first.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100