huggingface/trl

View on GitHub

Train transformer language models with reinforcement learning.

Stars: 19.3k
Forks: 3k
Open beginner issues: 0
Indexed issues: 129
Avg merge: 1d 20h
Merged PRs (30d): 194

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 16, 2026
Latest indexed: Sep 17, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Open All

129 open issues indexed

RLOOTrainer & PPOv2Trainer - Modify Name for W&B Logged Table Open

✨ enhancement 🏋 PPO 🏋 RLOO 🙋 help from community wanted

huggingface/trl#2045 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Supports of PPOTrainer / DPOTrainer for Qwen2Audio Open

✨ enhancement

huggingface/trl#2097 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
online DPO evaluation Open

🏋 Online DPO 🐛 bug

huggingface/trl#2228 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Packing in DPOTrainer Open

✨ enhancement 🏋 DPO 🚨 Important

huggingface/trl#2469 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
GKDTrainer + FSDP results in RuntimeError: Expected all tensors to be on the same device, but found at least two devices Open

🏋 GKD 🐛 bug

huggingface/trl#2580 · 9 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
`compute_metrics` in `GRPOTrainer` Open

✨ enhancement 🏋 GRPO

huggingface/trl#2959 · 3 comments · 13 reactions · 0 assignees · Indexed Sep 17, 2026
Numerical stability in KL's exp function Open

huggingface/trl#3015 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Sync Reference Model is incompatible with LoRA in GRPO Trainer Open

⚡ PEFT ⚡accelerate ✨ enhancement 🏋 GRPO

huggingface/trl#3108 · 5 comments · 3 reactions · 0 assignees · Indexed Sep 17, 2026
grpo init_communicator ipv6 can not reach Open

✨ enhancement 🏋 GRPO

huggingface/trl#3164 · 4 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
FP8 training Open

⚡accelerate ✨ enhancement

huggingface/trl#3399 · 13 comments · 2 reactions · 0 assignees · Indexed Sep 17, 2026
[GRPO] NCCL timeout when GRPO training with vllm, but get "ValueError: The decoder prompt (length 5951) is longer than the maximum model length of 4096. Make sure that `max_model_len` is no smaller than the number of text tokens. " Open

huggingface/trl#3433 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[GPG][new trainer] Add support to new `GPG` method Open

✨ enhancement

huggingface/trl#3472 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Packing sequences for memory efficiency in GRPO and other preference learning implementations Open

✨ enhancement 🏋 GRPO

huggingface/trl#3549 · 11 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
GRPOTrainer: Parallelism for updating named params Open

✨ enhancement 🏋 GRPO

huggingface/trl#3557 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature Request] support dynamic sampling for GRPO trainer Open

✨ enhancement 🏋 GRPO

huggingface/trl#3708 · 4 comments · 5 reactions · 1 assignee · Indexed Sep 17, 2026
Add activation checkpointing to DPO and GRPO Open

✨ enhancement 🏋 DPO 🏋 GRPO

huggingface/trl#3717 · 9 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Packing with VLMs Open

✨ enhancement 🏋 SFT

huggingface/trl#4339 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Replace torch image by vllm image in Docker Open

🏋 GRPO 📱 cli

huggingface/trl#4523 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
SFTTrainer: Logging Experts Statistics Open

⚡ PEFT ✨ enhancement 🏋 SFT

huggingface/trl#4611 · 9 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Simple Quick Start examples are great – but on what hardware? A note in the doc would be lovely. Open

huggingface/trl#4968 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[GRPO] AssertionError in vLLM when using QLoRA (4-bit) due to weight shape mismatch during merge_adapter Open

⚡ vLLM

huggingface/trl#4973 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Improper buggy SFT training of Llama-3.2-3B-Instruct. Open

🐛 bug 🚨 Important

huggingface/trl#5138 · 30 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Introduce a Minimal Backend Generation Interface Open

huggingface/trl#5193 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
Logging how vllm importance ratios are truncated/masked in GRPOTrainer Open

huggingface/trl#5231 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Multimodal generation hangs in vllm server mode with large images (works in colocate), manual resizing required Open

huggingface/trl#5361 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Video inputs for multimodal LLM Open

huggingface/trl#5373 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Possible NCCL ordering issue in TRL colocate mode with trainer TP + vLLM TP during tool-calling rollout Open

huggingface/trl#5382 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
# [Feature Request] GRPOTrainer: support stop-tool termination in agent training loop Open

huggingface/trl#5389 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Skip unnecessary no_grad forward pass for IS correction in on-policy vLLM training Open

huggingface/trl#5412 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
AsyncGRPO: support async tool calls in AsyncRolloutWorker Open

huggingface/trl#5444 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
TPOTrainer.evaluate() returns NaN eval_loss while training loss is finite Open

🐛 bug

huggingface/trl#5662 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Can gold_trainer support privileged context? Open

huggingface/trl#5741 · 4 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[AsyncGRPO] `AsyncGRPO` `add_response_schema` not gated on `self.tools` (diverges from `GRPOTrainer`) Open

huggingface/trl#5742 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
RDPO support in GRPOTRainer Open

huggingface/trl#5886 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Add adapter-only vLLM LoRA sync for GRPO and AsyncGRPO Open

huggingface/trl#5975 · 4 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
AsyncGRPOTrainer: vision-language (*ForConditionalGeneration) checkpoints can't be trained (weight-sync key mismatch) Open

⚡ vLLM 🏋 GRPO 👁️ VLM

huggingface/trl#6028 · 6 comments · 1 reaction · 1 assignee · Indexed Sep 17, 2026
Support router-based multi-teacher (MOPD) distillation in DistillationTrainer Open

huggingface/trl#6151 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature] Reasoning reward utilities: group-relative length-scaled accuracy (GRPO-LEAD) + graduated format reward Open

huggingface/trl#6221 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
GRPO + vLLM corrupts SmolVLM multimodal prompts from pre-expanded image tokens Open

huggingface/trl#6294 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[harbor/openreward] Adopt environment-owned reward (#6238) and dataset (#6349) Open

huggingface/trl#6354 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
Add UP (Unbounded Positive) loss as a GRPO loss_type Open

huggingface/trl#6407 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Remove bitsandbytes `_check_is_size` warning filter once the minimum required bitsandbytes is >= 0.50.0 Open

huggingface/trl#6447 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Remove triton `AnnAssign` warning filter once no supported torch resolves a triton without the fix (>= 3.8.0) Open

huggingface/trl#6466 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
`precompute_ref_log_probs=True` crashes under FSDP2 (device mismatch) Open

huggingface/trl#6470 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Add multi-GPU precompute test coverage for KTO Open

huggingface/trl#6471 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Implement single-rollout asynchronous optimization Open

huggingface/trl#6473 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Support image paths and URLs in vLLM generation Open

huggingface/trl#6519 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[CPO/SimPO] Truncation in CPOTrainer can silently produce empty completions -> NaN loss with loss_type="simpo" Open

huggingface/trl#6548 · 7 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Remove kernels-community/mamba-ssm autocast warning filter once the Hub kernel build stops calling `torch.get_autocast_gpu_dtype()` Open

huggingface/trl#6555 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Explore QAT support in TRL Open

huggingface/trl#6567 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026

huggingface/trl

Get new issues in your inbox