alibaba / alibaba/ROLL

[Bug] Multiple stability issues in RLVR pipeline: LoRA synchronization failure and recovery crash

Open
#436 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
312
Avg merge
1h 2m
Merged PRs (30d)
2

Description

## Describe the bug
During large-scale reinforcement learning (RLVR) training, we identified several critical issues affecting the stability and correctness of the ROLL framework:

1. LoRA Weight Inconsistency: When training with LoRA adapters, parameter updates are not correctly gathered and broadcasted from training nodes to rollout workers. This results in the inference phase using outdated base model weights.
2. State Recovery Failure: When resuming training, DynamicSamplingScheduler attempts to call get_next_dataset_item() before dataset_iter is properly initialized in the __init__ sequence, causing a crash.
3. DeepSpeed Group Initialization: DeepSpeed initialization fails if it receives an empty parameter group (common when freezing layers in LoRA).
4. Ray Metadata Overflow: Long-running sessions can exhaust the system's /tmp partition. There is no current way to redirect Ray's temporary directory.

## Logs

1 (DynamicSamplingScheduler) Traceback (most recent call last):
2 File ".../roll/distributed/scheduler/generate_scheduler.py", line 478, in __init__
3 self.get_next_dataset_item()
4 File ".../roll/distributed/scheduler/generate_scheduler.py", line 727, in get_next_dataset_item
5 if self.dataset_iter is None:
6 AttributeError: 'DynamicSamplingScheduler' object has no attribute 'dataset_iter'

DeepSpeed Error:
1 ValueError: optimizer got an empty parameter list

## Environment:
- Hardware: NVIDIA H200 Cluster
- Backend: DeepSpeed + Ray

Contributor guide

No contributing guide indexed for this repository

Research direction

The scheduler traceback points to roll/distributed/scheduler/generate_scheduler.py at lines 478 and 727; begin by reproducing the resume path and inspecting dataset_iter initialization. Then trace the LoRA synchronization, empty-parameter DeepSpeed initialization, and Ray temporary-directory handling from their existing entry points. Done means all four reported failures have verified fixes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.