NVIDIA-NeMo / NVIDIA-NeMo/RL

[Feature Request] Add configurable `resume_if_exists` flag to control automatic checkpoint resumption

Open
#345 4 comments 0 reactions 0 assignees View on GitHub
enhancement external t-checkpoint x-abeja
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

Hi team,

**Summary**
At the moment, the training scripts automatically resume from the latest checkpoint whenever `last_checkpoint_path` is found. This is convenient, but there are cases where I want to reuse the same `checkpoint_dir` while *not* resuming — i.e., I’d like to start training from scratch even if previous checkpoints are present.

**Why it matters**

* Experimentation often involves rerunning with the same output directory for ease of bookkeeping.
* Deleting or moving checkpoints manually is error‑prone, especially in shared storage or automated pipelines.
* Providing an opt‑in / opt‑out flag keeps the current convenience while adding flexibility.

**Where it shows up**
I noticed this in **GRPO**, but the same logic appears in **DPO** and **SFT**.
Example: in `nemo_rl/algorithms/grpo.py` around line 148 you immediately call

```python
last_checkpoint_path = checkpointer.get_latest_checkpoint_path()
```

without a way to override the behavior.

**Proposed interface**
Add a field under `checkpointing` (or similar) in the config, e.g.

```yaml
checkpointing:
resume_if_exists: true # default keeps current behaviour
```

Then gate the logic like:

```python
if master_config["checkpointing"].get("resume_if_exists", True):
last_checkpoint_path = checkpointer.get_latest_checkpoint_path()
else:
last_checkpoint_path = None
```

**Benefits**

* Zero disruption for existing users (default = current behaviour).
* Simple implementation; touches only the call site.
* Symmetric with NeMo’s `resume_if_exists` option in other training loops, so it feels familiar to users.

I’m not across the entire code‑base, so the exact wiring may differ, but I hope this illustrates the idea. Happy to refine the proposal or open a PR if you think it makes sense.

Thanks for your consideration!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.