CarperAI / CarperAI/trlx

RL related issues, code understanding , scores, rewards,ppo

Open
#326 1 comment 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
4.8k
Forks
487
PR merge metrics
No merged PRs in 30d

Description

### 🚀 The feature, motivation, and pitch

https://github.com/CarperAI/trlx/blob/b91da7b03d8e9fa0c0d6dce10a8f2611aca3013f/trlx/trainer/accelerate_ppo_trainer.py

(1)
In line 307,
scores = torch.tensor(
self.reward_fn(
samples=str_samples,
prompts=str_prompts,
outputs=str_outputs,
),
dtype=torch.float,
).to(device)

What's the meaning of scores and how it is calculated, what's self.reward_fn?
What role does scores play in the ppo RL framework?

(2)
In line 440, why let rewards[-1] adds scores[sample_idx]?
rewards = sample_kl_divergence_estimate
rewards[-1] += scores[sample_idx].cpu()

Thanks

### Alternatives

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.