[Refit] Recover safely after partial remote sparse refit failure
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
## Problem
Remote sparse refit applies payloads in place on rollout workers, but commits the
trainer-side XOR baseline only after every transport and receiver barrier succeeds.
If a sync fails after one or more receivers apply a payload, the receiver can be
partially advanced while the trainer baseline remains at the previous version.
Replaying the same relative XOR delta is unsafe because XOR is self-inverse and
would revert already-applied bytes.
PR #2444 makes this state fail loud by poisoning the synchronizer after a failed
sync. The only supported recovery is currently to reload rollout workers from a
known-good checkpoint and initialize a new communicator.
## Desired recovery
Provide an in-place transactional recovery path, for example by either:
- rolling every receiver back to the last committed weight version; or
- establishing a fresh common baseline and weight version on the trainer and all
receivers before allowing another sparse sync.
## Acceptance criteria
- A failure after partial receiver apply can be recovered without restarting the
training process.
- A retry cannot double-apply an XOR payload.
- Receiver and trainer weight versions are explicit and checked before commit.
- Failure-injection coverage includes partial S3 and ZeroMQ delivery.
Contributor guide
Research direction
Start by tracing the remote sparse refit synchronizer through trainer and receiver weight-version handling, then inspect the partial-delivery failure paths for S3 and ZeroMQ. Compare rollback and fresh-baseline recovery against the stated acceptance criteria, including failure-injection coverage. Done means recovery works in place, retries cannot double-apply XOR payloads, and versions are checked before commit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100