NVIDIA-NeMo / NVIDIA-NeMo/Gym

Allow explicit input files in the all-in-one gym eval run flow

Open
#3,501 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

community-request
Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
324

Description

Use cases, pain points, and background

The all-in-one gym eval run flow currently rejects --input and requires a configured dataset split. To run an already-prepared JSONL file, users have to start servers separately with gym env start, then collect rollouts with gym eval run --no-serve --input.

Supporting explicit inputs in the all-in-one flow would simplify running subsets or retrying selected tasks without adding a separate dataset configuration.

Description:

Allow gym eval run --input <prepared-tasks.jsonl> to handle server startup, rollout collection, and shutdown in one command. Any required dataset preparation still happens beforehand.

Design:

  • Update nemo_gym/rollout_collection.py to accept an explicit input path without requiring a split.
  • Update nemo_gym/cli/eval.py to validate the file before server startup and use it directly, bypassing preparation of configured dataset splits.
  • Give explicit input files precedence over configured splits, including when using a custom collection driver.
  • Preserve existing split preparation and cache reuse when no input file is supplied.
  • Update CLI help in nemo_gym/cli/main.py, the CLI reference under fern/, and the relevant unit tests.

Out of scope:

  • Automatically downloading or preparing the dataset supplied through --input.
  • Changing dataset formats, agent behavior, or verification logic.
  • Changing the existing --no-serve workflow.

Acceptance Criteria:

  • An already-prepared input file can be run without --split or --no-serve.
  • Missing or invalid file paths fail before servers start.
  • Explicit input takes precedence over a configured split.
  • Standard collection and custom collection drivers receive the supplied file.
  • Existing split preparation and cache reuse still work.
  • CLI help and documentation explain that input files must be prepared beforehand.
  • Regression tests and a live end-to-end smoke test pass.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nemo_gym/rollout_collection.py and nemo_gym/cli/eval.py to trace input validation, split preparation, server startup, and collection-driver selection. Then inspect help in nemo_gym/cli/main.py, the CLI reference under fern/, and the relevant unit tests. Done means explicit prepared files work for standard and custom drivers, invalid paths fail before startup, existing split and cache behavior remains intact, and the regression and live smoke tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, documentation, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.