NVIDIA-NeMo / NVIDIA-NeMo/Gym

NeMo RL + Gym integration: dynamic sampling

Open
#199 0 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

needs-design training
Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 21h
Merged PRs (30d)
318

Description

Use cases, pain points, and background
We want to only train on mixed-reward examples to improve training efficiency.

Description:
Implement dynamic sampling. As soon as a group finishes, we check it for mixed rewards and kick off another group if necessary until the entire batch is full.

Design:
Probably need to touch a few files, but most likely we need to shuffle some responsibility around in the rollout + Gym logic.

Need to figure out how to yield one prompt at a time from the train dataloader and see how that works. Need some investigation into torch StatefulDataloader and possibly the collation function.

May need to do one collation at the end.

Out of scope:

Acceptance Criteria:

  • Implementation of the above + tests in NeMo RL.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.