PPO efficiency improvement by not recomputing the advantage in Trainer
Open
bug
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 487
- PR merge metrics
- No merged PRs in 30d
Description
### 🐛 Describe the bug
Currently, advantage estimation is performed in Trainer (see [this](https://github.com/CarperAI/trlx/blob/94adc058d3de3f2645255dc2e2ca06c1a5a46cc9/trlx/trainer/accelerate_ppo_trainer.py#L114)). The per-step advantages however only depend on the rollout policy, and thus these quantities can be precomputed during rollout, instead of recomputed during PPO updates.
### Which trlX version are you using?
_No response_
### Additional system and package information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.