Sync Reference Model is incompatible with LoRA in GRPO Trainer
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Feature request
## Issue Description
In the current implementation of `GRPOTrainer`, there is a conflict between using LoRA (PEFT adapters) and the `sync_ref_model` parameter. When LoRA is enabled, the reference model synchronization mechanism is completely bypassed, preventing users from having a properly synchronized reference model with LoRA updates.
## Current Behavior
Currently, when LoRA is used:
1. The trainer sets `self.ref_model = None` in the initialization
2. For reference model computations, the trainer temporarily disables the adapter instead of using a separate reference model
3. If `sync_ref_model=True` is set, the added callback is ineffective since `ref_model` is `None`
This means that users cannot benefit from both LoRA's parameter efficiency and proper reference model synchronization at the same time.
## Expected Behavior
Users should be able to use LoRA with a properly synchronized reference model. The expected behavior would be:
1. When using LoRA with `sync_ref_model=True`:
- Create a separate reference model with its own LoRA adapter
- Periodically synchronize the policy model's LoRA weights to the reference model
2. This would allow the reference model to properly track the policy model's learning progress, which is important for algorithms like GRPO that use the KL divergence between policy and reference models.
## Additional Information
- TRL version: 0.15.2
- Using PEFT: Yes
- Using accelerate: Yes
This issue affects GRPO training when trying to use LoRA with reference model synchronization, which is a common setup for efficiently fine-tuning large language models.
### Motivation
Users can benefit from both PEFT and TR-DPO.
### Your contribution
None
Contributor guide
Research direction
Start in the GRPOTrainer implementation by tracing LoRA initialization, the self.ref_model assignment, reference-model computations, and the sync_ref_model callback. Verify how PEFT adapters are created and synchronized, then confirm that sync_ref_model=True keeps a separate reference adapter updated while reference computations continue to work with LoRA.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100