deepspeedai / deepspeedai/DeepSpeedExamples
PPO training unable to reproduce the training log provided
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
Many thanks for you guys great work! I am following your provided training code and scritps trying to reproduce the actor-opt-1.3b-critic-opt-350m results. However, the reward curve is always lies in around ~-4, and it gets bad reward at the very begining of the training. I have run some qualitative study on the 0 step collected experiences:
Is this behaviour expected? Is this issue related to tokenizer or should we mask out what is after the "<|endoftext|>" in the model generated results?
I have compared the training logs and do some very simple qualitative study on step1 and step2 training, and I do not find any obvious bugs.
Many thanks in advance!!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.