huggingface / huggingface/trl

Sync Reference Model is incompatible with LoRA in GRPO Trainer

Open
#3,108 5 comments 3 reactions 0 assignees View on GitHub
⚡ PEFT ⚡accelerate ✨ enhancement 🏋 GRPO
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Feature request

## Issue Description

In the current implementation of `GRPOTrainer`, there is a conflict between using LoRA (PEFT adapters) and the `sync_ref_model` parameter. When LoRA is enabled, the reference model synchronization mechanism is completely bypassed, preventing users from having a properly synchronized reference model with LoRA updates.

## Current Behavior

Currently, when LoRA is used:

1. The trainer sets `self.ref_model = None` in the initialization
2. For reference model computations, the trainer temporarily disables the adapter instead of using a separate reference model
3. If `sync_ref_model=True` is set, the added callback is ineffective since `ref_model` is `None`

This means that users cannot benefit from both LoRA's parameter efficiency and proper reference model synchronization at the same time.

## Expected Behavior

Users should be able to use LoRA with a properly synchronized reference model. The expected behavior would be:

1. When using LoRA with `sync_ref_model=True`:
- Create a separate reference model with its own LoRA adapter
- Periodically synchronize the policy model's LoRA weights to the reference model

2. This would allow the reference model to properly track the policy model's learning progress, which is important for algorithms like GRPO that use the KL divergence between policy and reference models.

## Additional Information

- TRL version: 0.15.2
- Using PEFT: Yes
- Using accelerate: Yes

This issue affects GRPO training when trying to use LoRA with reference model synchronization, which is a common setup for efficiently fine-tuning large language models.

### Motivation

Users can benefit from both PEFT and TR-DPO.

### Your contribution

None

Contributor guide

Open the contributing guide

Research direction

Start in the GRPOTrainer implementation by tracing LoRA initialization, the self.ref_model assignment, reference-model computations, and the sync_ref_model callback. Verify how PEFT adapters are created and synchronized, then confirm that sync_ref_model=True keeps a separate reference adapter updated while reference computations continue to work with LoRA.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.