NeMo RL + Gym integration: dynamic sampling
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 318
Description
Use cases, pain points, and background
We want to only train on mixed-reward examples to improve training efficiency.
Description:
Implement dynamic sampling. As soon as a group finishes, we check it for mixed rewards and kick off another group if necessary until the entire batch is full.
Design:
Probably need to touch a few files, but most likely we need to shuffle some responsibility around in the rollout + Gym logic.
Need to figure out how to yield one prompt at a time from the train dataloader and see how that works. Need some investigation into torch StatefulDataloader and possibly the collation function.
May need to do one collation at the end.
Out of scope:
Acceptance Criteria:
- Implementation of the above + tests in NeMo RL.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.