NVIDIA-NeMo / NVIDIA-NeMo/RL

Catch config/data validation errors during preflight (or via a lightweight dry-run) instead of mid-run

Open
#2,783 1 comment 0 reactions 1 assignee Claimed by @terrykong View on GitHub
Feature
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

### Summary

In NeMo-RL + NeMo-Gym, validation errors tend to surface *late* in a run rather than during preflight. Because failures can land after model bring-up, a full start-of-training validation epoch, and/or full train rollouts, a single misconfiguration can cost 30+ minutes before it's caught.

### Examples encountered

1. **`make_sequence_length_divisible_by` failure** - killed a run only *after* completing a full validation epoch (val-at-start enabled) and full train rollouts.
2. **NeMo-Gym data validation** - a run failed because the `license` field in the data `.jsonl` mapped to a value NeMo-Gym didn't recognize. This surfaces only after ~10 minutes of NeMo-RL bring-up.

### Proposal

- Move validations like the above to the **start of preflight** so they fail fast.
- Add a lightweight **`--dry-run`** (or equivalent) option to NeMo-RL that:
- runs config / padding validations (e.g. sequence-length divisibility) without launching a full run, and
- passes through to **NeMo-Gym's dry-run** to catch data-validation issues (e.g. unrecognized field values) up front.

### Current workaround

Smoke-testing end-to-end with a tiny model (e.g. 3 random-weight layers) catches script-level breakage, but still takes a couple of minutes of setup and doesn't substitute for fast, targeted preflight validation.

### Benefit

Tighter feedback loop; avoids sinking 10–30+ minutes into runs that fail on trivially detectable config/data errors.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.