RL related issues, code understanding , scores, rewards,ppo
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 487
- PR merge metrics
- No merged PRs in 30d
Description
### 🚀 The feature, motivation, and pitch
https://github.com/CarperAI/trlx/blob/b91da7b03d8e9fa0c0d6dce10a8f2611aca3013f/trlx/trainer/accelerate_ppo_trainer.py
(1)
In line 307,
scores = torch.tensor(
self.reward_fn(
samples=str_samples,
prompts=str_prompts,
outputs=str_outputs,
),
dtype=torch.float,
).to(device)
What's the meaning of scores and how it is calculated, what's self.reward_fn?
What role does scores play in the ppo RL framework?
(2)
In line 440, why let rewards[-1] adds scores[sample_idx]?
rewards = sample_kl_divergence_estimate
rewards[-1] += scores[sample_idx].cpu()
Thanks
### Alternatives
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.