Catch config/data validation errors during preflight (or via a lightweight dry-run) instead of mid-run
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
### Summary
In NeMo-RL + NeMo-Gym, validation errors tend to surface *late* in a run rather than during preflight. Because failures can land after model bring-up, a full start-of-training validation epoch, and/or full train rollouts, a single misconfiguration can cost 30+ minutes before it's caught.
### Examples encountered
1. **`make_sequence_length_divisible_by` failure** - killed a run only *after* completing a full validation epoch (val-at-start enabled) and full train rollouts.
2. **NeMo-Gym data validation** - a run failed because the `license` field in the data `.jsonl` mapped to a value NeMo-Gym didn't recognize. This surfaces only after ~10 minutes of NeMo-RL bring-up.
### Proposal
- Move validations like the above to the **start of preflight** so they fail fast.
- Add a lightweight **`--dry-run`** (or equivalent) option to NeMo-RL that:
- runs config / padding validations (e.g. sequence-length divisibility) without launching a full run, and
- passes through to **NeMo-Gym's dry-run** to catch data-validation issues (e.g. unrecognized field values) up front.
### Current workaround
Smoke-testing end-to-end with a tiny model (e.g. 3 random-weight layers) catches script-level breakage, but still takes a couple of minutes of setup and doesn't substitute for fast, targeted preflight validation.
### Benefit
Tighter feedback loop; avoids sinking 10–30+ minutes into runs that fail on trivially detectable config/data errors.
Contributor guide
Assessment
This issue has not been assessed yet.