epic: improved environment discovery with webapp
Open
Nobody has claimed this yet.
- #918 by @cmunley1 — closed without merging
CLI
community-request
Environment Discoverability
needs-design
usability
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 318
Description
Use cases, pain points, and background
NeMo Gym currently has 40+ environments with limited discovery, comparison, or quality signaling mechanism. Current state:
- A README table auto-generated from YAML configs (columns: name, domain, description, value, train/validation availability, license, config link, dataset link)
- Some profiling artifacts in individual server data/ directories
- Data for use in training is not immediately available for use in training, an additional step is required to pull the data from hugging face
- Server READMEs are mixed in quality and completeness, largely containing unstructured data
Users today cannot easily answer:
- "Which environments exist for math training?" (requires scanning README table)
- "What does this environment actually do?" (requires reading each server's README or inspecting code)
- "How hard is this environment? What should I expect my model to score?" (requires running profiling yourself)
- "Can I use this with my agent?" (requires reading the code)
- "Is this environment production-quality?" (we do validation runs but don’t share these publicly currently)
- “Which environments have data ready to use?” (README doesn’t differentiate between ones that have data readily available, which have data generation scripts available, which may have upcoming datasets not yet released)
Description
Requirement Categories
- Discovery: What environments exist for my use case?
- Understanding: What does this environment actually do?
- Quality Signal: Can I trust this environment?
- Difficulty Calibration: Is this the right difficulty for my model?
- Compatibility: Will this work with my setup?
- Getting Started: How do I run this?
- Authoring: How do I register my environment?
| Priority | Category | Requirement |
|---|---|---|
| P1 | Discovery | PR1.2: Browse all environments in an interactive web registry |
| P1 | Discovery | PR1.3: Filter by domain (math, coding, agent, knowledge, etc.) |
| P1 | Discovery | PR1.4: Filter by interaction pattern (single-turn, multi-step, multi-turn, agentic sandbox) |
| P1 | Discovery | PR1.5: Search by free-text (name, description, tags) |
| P1 | Discovery | PR1.6: Share link to specific environment |
| P1 | Discovery | PR1.7: Filter by practical constraints (compute requirements, external deps, Docker/GPU) |
| P2 | Discovery | PR1.8: Aggregate stats of NeMo Gym environments (e.g. total count, domain distribution, quality distribution) |
| P1 | Understanding | PR2.1: Standardized summary for each environment (name, description, domain, value proposition) |
| P1 | Understanding | PR2.2: Interaction pattern formally described (how the agent-environment loop works) |
| P1 | Understanding | PR2.3: Available tools listed with their schemas (what can the agent call?) |
| P1 | Understanding | PR2.4: Verification methodology clearly described detailing how reward is assigned |
| P1 | Understanding | PR2.5: Data summarized (dataset sizes per split, license, source) |
| P1 | Understanding | PR2.6: Example trajectories or rollouts viewable |
| P1 | Quality | PR3.1: Source repository and upstream attribution visible |
| P1 | Quality | PR3.2: Authorship and last-updated date visible |
| P1 | Quality | PR3.3: Verification criteria transparent and well-defined |
| P1 | Calibration | PR4.1: Reference model performance visible per environment (pass@1, pass@k) |
| P1 | Calibration | PR4.2: Compare model performance across environments (relative difficulty) |
| P2 | Calibration | PR4.3: Filter environments by aggregate difficulty relative to a model class |
| P1 | Calibration | PR4.4: Profiling results include enough context to be reproducible (model, agent, temperature, num_repeats, date) |
| P0 | Compatibility | PR5.1: Each environment declares compatible agent(s) |
| P0 | Compatibility | PR5.2: Each environment declares supported resource server(s) |
| P0 | Compatibility | PR5.3: Each environment declares compute/infrastructure requirements (beyond requirements.txt also including containers, external services etc) |
| P0 | Compatibility | PR5.5: Known limitations or training gotchas documented |
| P0 | Getting Started | PR6.1: Quick-start snippet per environment (ng_run + ng_collect_rollouts commands) |
| P1 | Getting Started | PR6.2: Links to config files and documentation |
| P1 | Getting Started | PR6.3: Links to source code |
| P2 | Getting Started | PR6.4: External dependencies have setup instructions or are auto-installed |
| P0 | Authoring | PR7.1: Registry metadata defined alongside existing YAML config |
| P0 | Authoring | PR7.2: Metadata validated automatically (pre-commit or CI) |
| P1 | Authoring | PR7.3: Profiling results can be added incrementally after initial registration |
| P1 | Authoring | PR7.4: Schema extensible for benchmark-specific metadata without breaking core spec |
Design:
What files should be touched? What logic should be written?
Out of scope:
What are some items that this issue could be mistaken to cover that this issue should explicitly NOT cover?
Acceptance Criteria:
- Individual items that need to be finished in order for this issue to be considered completed
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation files or tests are named. Start by inventorying the auto-generated README table, YAML configs, server README files, and server data/ directories, then clarify which requirement category and acceptance criteria are in scope. Done should be defined as a bounded set of completed requirements rather than the full web registry proposal.
Written by the indexing model from the issue text.
Assessment
- Domain
- web-dev
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100