Allow explicit input files in the all-in-one gym eval run flow
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 324
Description
Use cases, pain points, and background
The all-in-one gym eval run flow currently rejects --input and requires a configured dataset split. To run an already-prepared JSONL file, users have to start servers separately with gym env start, then collect rollouts with gym eval run --no-serve --input.
Supporting explicit inputs in the all-in-one flow would simplify running subsets or retrying selected tasks without adding a separate dataset configuration.
Description:
Allow gym eval run --input <prepared-tasks.jsonl> to handle server startup, rollout collection, and shutdown in one command. Any required dataset preparation still happens beforehand.
Design:
- Update
nemo_gym/rollout_collection.pyto accept an explicit input path without requiring a split. - Update
nemo_gym/cli/eval.pyto validate the file before server startup and use it directly, bypassing preparation of configured dataset splits. - Give explicit input files precedence over configured splits, including when using a custom collection driver.
- Preserve existing split preparation and cache reuse when no input file is supplied.
- Update CLI help in
nemo_gym/cli/main.py, the CLI reference underfern/, and the relevant unit tests.
Out of scope:
- Automatically downloading or preparing the dataset supplied through
--input. - Changing dataset formats, agent behavior, or verification logic.
- Changing the existing
--no-serveworkflow.
Acceptance Criteria:
- An already-prepared input file can be run without
--splitor--no-serve. - Missing or invalid file paths fail before servers start.
- Explicit input takes precedence over a configured split.
- Standard collection and custom collection drivers receive the supplied file.
- Existing split preparation and cache reuse still work.
- CLI help and documentation explain that input files must be prepared beforehand.
- Regression tests and a live end-to-end smoke test pass.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in nemo_gym/rollout_collection.py and nemo_gym/cli/eval.py to trace input validation, split preparation, server startup, and collection-driver selection. Then inspect help in nemo_gym/cli/main.py, the CLI reference under fern/, and the relevant unit tests. Done means explicit prepared files work for standard and custom drivers, invalid paths fail before startup, existing split and cache behavior remains intact, and the regression and live smoke tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, documentation, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100