NVIDIA-NeMo / NVIDIA-NeMo/RL

Track the Automodel save_consolidated process-group initialization fix

Open
#3,893 0 comments 0 reactions 1 assignee Claimed by @jQizhang View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

This issue tracks a follow-up PR to #3498. When `save_consolidated: true` is applied after the Automodel `Checkpointer` is initialized, the dedicated Gloo process group is not created, so async consolidation may fall back to the default NCCL group and hang.

The follow-up PR should fix the Checkpointer lifecycle and add multi-rank validation.

Related: #3624

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.