NVIDIA-NeMo / NVIDIA-NeMo/Gym

Rollout dataset type implementation and validation

Open
#14 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

core-infra
Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 21h
Merged PRs (30d)
318

Description

We should have explicit support in Gym for rollout datasets, datasets that are the result of running datasets through NeMo Gym agents. This would help efforts related to reward model training and SFT.

Specifically right now we have train, validation, and example dataset types. We should add a new rollout dataset type and then doing the corresponding validation similar to the train preparation. Rollout dataset validation validates that every row in the dataset is a valid instance of BaseVerifyResponse (link) which includes the responses create params, response, and reward.

Thanks to @HeyyyyyyG for the raise!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing train, validation, and example dataset type handling and the BaseVerifyResponse definition in nemo_gym/base_resources_server.py. Trace the train preparation validation path, then add the rollout type so each row is checked for response create parameters, response, and reward; done means valid rollout datasets are accepted and invalid rows are rejected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.