[Feature Request] Add configurable `resume_if_exists` flag to control automatic checkpoint resumption
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
Hi team,
**Summary**
At the moment, the training scripts automatically resume from the latest checkpoint whenever `last_checkpoint_path` is found. This is convenient, but there are cases where I want to reuse the same `checkpoint_dir` while *not* resuming — i.e., I’d like to start training from scratch even if previous checkpoints are present.
**Why it matters**
* Experimentation often involves rerunning with the same output directory for ease of bookkeeping.
* Deleting or moving checkpoints manually is error‑prone, especially in shared storage or automated pipelines.
* Providing an opt‑in / opt‑out flag keeps the current convenience while adding flexibility.
**Where it shows up**
I noticed this in **GRPO**, but the same logic appears in **DPO** and **SFT**.
Example: in `nemo_rl/algorithms/grpo.py` around line 148 you immediately call
```python
last_checkpoint_path = checkpointer.get_latest_checkpoint_path()
```
without a way to override the behavior.
**Proposed interface**
Add a field under `checkpointing` (or similar) in the config, e.g.
```yaml
checkpointing:
resume_if_exists: true # default keeps current behaviour
```
Then gate the logic like:
```python
if master_config["checkpointing"].get("resume_if_exists", True):
last_checkpoint_path = checkpointer.get_latest_checkpoint_path()
else:
last_checkpoint_path = None
```
**Benefits**
* Zero disruption for existing users (default = current behaviour).
* Simple implementation; touches only the call site.
* Symmetric with NeMo’s `resume_if_exists` option in other training loops, so it feels familiar to users.
I’m not across the entire code‑base, so the exact wiring may differ, but I hope this illustrates the idea. Happy to refine the proposal or open a PR if you think it makes sense.
Thanks for your consideration!
Contributor guide
Assessment
This issue has not been assessed yet.