NVIDIA-NeMo / NVIDIA-NeMo/Gym

bug: data availability with each environment unclear

Open
#982 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

community-request
Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 21h
Merged PRs (30d)
318

Description

Describe the bug

The README table has many resources servers listed with "-" under the dataset column, users have expressed confusion whether data is available or not, and if/when it is coming.

  • some environments we procedurally generate data, in these cases there is a script(s) in the resources folder directory for that environment, e.g. reasoning gym
  • some of our environments are interfaces to other environment libraries that have their own collections of environments and data, e.g. open env
  • some may have associated datasets but missing the link in the README, e.g. equivalence LLM judge can be used with this openQA dataset
  • some environments we may have data in progress but not yet released

Steps/Code to reproduce bug

View README table

Expected behavior

More precision around data availability, such that users are able to readily use environments where data is available or can be generated with existing scripts, vs which environments require their own data preparation

Configs
NeMo Gym v0.2.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the README table and inspect the resources-folder scripts for environments such as reasoning gym. Compare cases like Open Env, equivalence LLM judge, and the referenced OpenQA dataset to determine how availability is currently represented. Done means the README distinguishes generated data, external collections, linked datasets, and data that is not yet released or requires user preparation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.