NVIDIA-NeMo / NVIDIA-NeMo/Automodel
qwen3-32b fails OOM error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Describe the bug
Training crashes with CUDA out of memory at step 635 during loss computation (logits.float()). Tried to allocate 50.51 GiB but only 44.84 GiB was free on GPU 1. Grad norm also spikes sharply (22 → 193–198) right before the crash.
Steps/Code to reproduce bug
- Run finetune.py from /opt/Automodel/examples/llm_finetune/
- Training runs normally until step ~635
- Crash occurs during loss calculation in masked_ce.py line 74
Expected behavior
Training completes without OOM errors.
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 50.51 GiB. GPU 1 total: 178.36 GiB | Free: 44.84 GiB | PyTorch allocated: 130.43 GiB → masked_ce.py line 74: logits = logits.float()
Additional context
Multi-GPU (4 GPUs), crash on rank1
Possible gradient explosion before OOM
Try: PYTORCH_CUDA_ALLOC_CONF=expandable_segments=True, gradient clipping, or keeping logits in bf16
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the run from /opt/Automodel/examples/llm_finetune/finetune.py and inspect the loss calculation at masked_ce.py line 74. Compare the rank 1 memory usage and gradient norm near step 635, including the supplied qwen3-8b log. Done means the qwen3-32b multi-GPU training completes without the CUDA OOM during loss computation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100