CarperAI / CarperAI/trlx

Confused by config parameters among total_steps, epochs, batch_size and num_rollouts

Open
#425 0 comments 5 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
4.8k
Forks
487
PR merge metrics
No merged PRs in 30d

Description

### 📚 The doc issue

TrainConfig has some general explanation about some of the parameters. But after running the [ppo_hh.py](https://github.com/CarperAI/trlx/blob/main/examples/hh/ppo_hh.py), i got confused.
1. ppo_hh.py set total_steps and epochs while it seems epochs was ignored in the training.
2. How many data will a step consumed? num_rollout or batch_size?
3. In my understanding, every epoch uses num_rollout examples. They are used for generation in a chunk_size batch. Then every step use batch_size to update the policy model. Am i right?

Thank you in advance for answering all the questions.

### Suggest a potential alternative/fix

Maybe there should be some more detailed description about these parameters?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.