CarperAI/trlx

View on GitHub

A repo for distributed training of language models with Reinforcement Learning via Human Feedback (RLHF)

Stars: 4.8k
Forks: 487
Open beginner issues: 0
Indexed issues: 86

Dominant language: Python
License: MIT
Last GitHub push: Jan 8, 2024
Latest indexed: Sep 14, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: good first issue
PR merge metrics: No merged PRs in 30d

Open All

86 open issues indexed

Gradient metrics Open

feature request

CarperAI/trlx#27 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Self Play Open

feature request

CarperAI/trlx#28 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Add Jax support Open

feature request

CarperAI/trlx#54 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
How we can improve the documentation for beginners Open

documentation

CarperAI/trlx#65 · 6 comments · 5 reactions · 0 assignees · Indexed Sep 14, 2026
How to attribute reward to multiple model runs in the same trajectory with PPO Open

feature request

CarperAI/trlx#69 · 7 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
How to attribute different rewards to parts of the same rollout with PPO? Open

feature request

CarperAI/trlx#79 · 5 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
Best-of-n sampling for rollouts Open

feature request

CarperAI/trlx#166 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 14, 2026
T5 FLAN 11B Config for a consumer GPU Open

feature request

CarperAI/trlx#173 · 3 comments · 7 reactions · 0 assignees · Indexed Sep 14, 2026
Improve Documentation Process Open

feature request

CarperAI/trlx#176 · 0 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
Add CD to publish PyPi Package Open

feature request

CarperAI/trlx#199 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
PPO efficiency improvement by not recomputing the advantage in Trainer Open

bug

CarperAI/trlx#202 · 4 comments · 3 reactions · 0 assignees · Indexed Sep 14, 2026
Remove yaml config files Open

feature request

CarperAI/trlx#223 · 4 comments · 7 reactions · 1 assignee · Indexed Sep 14, 2026
ModelConfig.tokenizer_path and TrainConfig.save_best Open

documentation

CarperAI/trlx#232 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Add Speculative Sampling Open

feature request research

CarperAI/trlx#273 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Remove dynamic imports Open

feature request

CarperAI/trlx#277 · 4 comments · 2 reactions · 1 assignee · Indexed Sep 14, 2026
Add tensor padding options to improve hardware utilization Open

contributions welcome feature request good first issue

CarperAI/trlx#284 · 8 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
Give a way to pass data that doesn't involve loading the whole prompt dataset to memory Open

feature request

CarperAI/trlx#304 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Colab + Accelerate Integration issue Open

bug

CarperAI/trlx#317 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Collapse reference+learner hydra heads when using LoRa Open

contributions welcome feature request good first issue

CarperAI/trlx#320 · 6 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
TPU Integration Open

feature request

CarperAI/trlx#323 · 3 comments · 1 reaction · 1 assignee · Indexed Sep 14, 2026
RL related issues, code understanding , scores, rewards,ppo Open

feature request

CarperAI/trlx#326 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Generic Flan-T5 RL training Open

CarperAI/trlx#327 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
SHP reward model Open

feature request

CarperAI/trlx#336 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Minimum Risk Training support Open

feature request

CarperAI/trlx#354 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Add support XGLM model Open

feature request

CarperAI/trlx#379 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
Flant-t5-large Deepspeed OVERFLOW! issues + bad outputs after trlx training Open

bug

CarperAI/trlx#383 · 8 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
@LouisCastricato Yes, the idea is to give a fully reproducible script for the anyone to train it by themselves and validate. Open

CarperAI/trlx#393 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 14, 2026
!deepspeed examples/summarize_rlhf/sft/train_gptj_summarize.py is failing Open

CarperAI/trlx#410 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Question about trainer.save_pretrained Open

feature request

CarperAI/trlx#412 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Well-known RMs and eval utilities Open

feature request

CarperAI/trlx#415 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Confused by config parameters among total_steps, epochs, batch_size and num_rollouts Open

documentation

CarperAI/trlx#425 · 0 comments · 5 reactions · 0 assignees · Indexed Sep 14, 2026
Training problem for the GPT-j RLHF example Open

CarperAI/trlx#428 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Migrate to `peft` from `opendelta` for parameter efficient tuning methods Open

feature request

CarperAI/trlx#434 · 6 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
FLAN-T5-small model run sentiments.ipynb demo gives so so many ""using bos_token ,but it is not set yet" lines" Open

bug

CarperAI/trlx#436 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Loss should be averaged over all samples instead of tokens Open

bug

CarperAI/trlx#452 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
reproduce trlx example on wandb Open

bug

CarperAI/trlx#462 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Release examples of training stablevicuna Open

feature request

CarperAI/trlx#463 · 3 comments · 4 reactions · 0 assignees · Indexed Sep 14, 2026
Precompute logprobs, values Open

CarperAI/trlx#467 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Issues in experience rollout generation with FLAN-T5 Open

bug

CarperAI/trlx#468 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
missing pad_token error when using GPT2Chinese Open

bug

CarperAI/trlx#469 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Support logging for non-scalar metrics Open

feature request

CarperAI/trlx#493 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Support for CodeGenForCausalLM Open

feature request

CarperAI/trlx#495 · 3 comments · 3 reactions · 0 assignees · Indexed Sep 14, 2026
Error loading T5 model checkpoint for inference after training with ILQL Open

bug

CarperAI/trlx#500 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Use tiny models for the tests Open

feature request

CarperAI/trlx#502 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
About the weight of word embedding being nan Open

bug

CarperAI/trlx#503 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Direct Policy Optimization Open

feature request

CarperAI/trlx#504 · 4 comments · 4 reactions · 0 assignees · Indexed Sep 14, 2026
Add support for safetensors Open

feature request

CarperAI/trlx#505 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
8-bit inference Open

bug

CarperAI/trlx#512 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
support base model + multi adapter for actor, critic, ref and reward model Open

feature request

CarperAI/trlx#518 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
ppo using GLM2-6b as a backbone? Open

feature request

CarperAI/trlx#523 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026

CarperAI/trlx

Get new issues in your inbox