NVIDIA-NeMo / NVIDIA-NeMo/RL

[Refit] Support remote sparse refit in PPO and distillation

Open
#3,275 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

## Problem

The remote S3/ZeroMQ sparse-refit lifecycle introduced by PR #2444 is wired only
through GRPO setup: validation, baseline startup, synchronizer attachment, and
weight-version commit all live on that path. PPO and distillation currently use
the existing dense refit path.

PR #2444 adds a setup-time error when `policy.generation.refit_transport` is set
for PPO or distillation so the option cannot silently no-op or fall back to dense
NCCL.

## Desired support

Extend remote sparse refit to PPO and distillation with the same lifecycle and
correctness contract as GRPO.

## Acceptance criteria

- Both algorithms validate the remote transport configuration before allocating
workers.
- Baseline construction overlaps independent rollout-engine initialization.
- The sparse synchronizer is attached before the first refit.
- Failed syncs obey the same terminal/recovery contract as GRPO.
- End-to-end tests compare the sparse path with each algorithm's dense control.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the GRPO setup and remote sparse-refit lifecycle introduced by PR #2444, then compare the PPO and distillation setup paths. Verify validation, baseline initialization overlap, synchronizer attachment, and failed-sync recovery, and add end-to-end comparisons between each algorithm's sparse and dense paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.