[Bug] Multiple stability issues in RLVR pipeline: LoRA synchronization failure and recovery crash
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 312
- Avg merge
- 1h 2m
- Merged PRs (30d)
- 2
Description
## Describe the bug
During large-scale reinforcement learning (RLVR) training, we identified several critical issues affecting the stability and correctness of the ROLL framework:
1. LoRA Weight Inconsistency: When training with LoRA adapters, parameter updates are not correctly gathered and broadcasted from training nodes to rollout workers. This results in the inference phase using outdated base model weights.
2. State Recovery Failure: When resuming training, DynamicSamplingScheduler attempts to call get_next_dataset_item() before dataset_iter is properly initialized in the __init__ sequence, causing a crash.
3. DeepSpeed Group Initialization: DeepSpeed initialization fails if it receives an empty parameter group (common when freezing layers in LoRA).
4. Ray Metadata Overflow: Long-running sessions can exhaust the system's /tmp partition. There is no current way to redirect Ray's temporary directory.
## Logs
1 (DynamicSamplingScheduler) Traceback (most recent call last):
2 File ".../roll/distributed/scheduler/generate_scheduler.py", line 478, in __init__
3 self.get_next_dataset_item()
4 File ".../roll/distributed/scheduler/generate_scheduler.py", line 727, in get_next_dataset_item
5 if self.dataset_iter is None:
6 AttributeError: 'DynamicSamplingScheduler' object has no attribute 'dataset_iter'
DeepSpeed Error:
1 ValueError: optimizer got an empty parameter list
## Environment:
- Hardware: NVIDIA H200 Cluster
- Backend: DeepSpeed + Ray
Contributor guide
No contributing guide indexed for this repository
Research direction
The scheduler traceback points to roll/distributed/scheduler/generate_scheduler.py at lines 478 and 727; begin by reproducing the resume path and inspecting dataset_iter initialization. Then trace the LoRA synchronization, empty-parameter DeepSpeed initialization, and Ray temporary-directory handling from their existing entry points. Done means all four reported failures have verified fixes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100