NVIDIA-NeMo / NVIDIA-NeMo/RL

[Refit] Recover safely after partial remote sparse refit failure

Open
#3,274 0 comments 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

## Problem

Remote sparse refit applies payloads in place on rollout workers, but commits the
trainer-side XOR baseline only after every transport and receiver barrier succeeds.
If a sync fails after one or more receivers apply a payload, the receiver can be
partially advanced while the trainer baseline remains at the previous version.
Replaying the same relative XOR delta is unsafe because XOR is self-inverse and
would revert already-applied bytes.

PR #2444 makes this state fail loud by poisoning the synchronizer after a failed
sync. The only supported recovery is currently to reload rollout workers from a
known-good checkpoint and initialize a new communicator.

## Desired recovery

Provide an in-place transactional recovery path, for example by either:

- rolling every receiver back to the last committed weight version; or
- establishing a fresh common baseline and weight version on the trainer and all
receivers before allowing another sparse sync.

## Acceptance criteria

- A failure after partial receiver apply can be recovered without restarting the
training process.
- A retry cannot double-apply an XOR payload.
- Receiver and trainer weight versions are explicit and checked before commit.
- Failure-injection coverage includes partial S3 and ZeroMQ delivery.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the remote sparse refit synchronizer through trainer and receiver weight-version handling, then inspect the partial-delivery failure paths for S3 and ZeroMQ. Compare rollback and fresh-baseline recovery against the stated acceptance criteria, including failure-injection coverage. Done means recovery works in place, retries cannot double-apply XOR payloads, and versions are checked before commit.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.