bug: data availability with each environment unclear
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 318
Description
Describe the bug
The README table has many resources servers listed with "-" under the dataset column, users have expressed confusion whether data is available or not, and if/when it is coming.
- some environments we procedurally generate data, in these cases there is a script(s) in the resources folder directory for that environment, e.g. reasoning gym
- some of our environments are interfaces to other environment libraries that have their own collections of environments and data, e.g. open env
- some may have associated datasets but missing the link in the README, e.g. equivalence LLM judge can be used with this openQA dataset
- some environments we may have data in progress but not yet released
Steps/Code to reproduce bug
View README table
Expected behavior
More precision around data availability, such that users are able to readily use environments where data is available or can be generated with existing scripts, vs which environments require their own data preparation
Configs
NeMo Gym v0.2.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the README table and inspect the resources-folder scripts for environments such as reasoning gym. Compare cases like Open Env, equivalence LLM judge, and the referenced OpenQA dataset to determine how availability is currently represented. Done means the README distinguishes generated data, external collections, linked datasets, and data that is not yet released or requires user preparation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100