Lightning-AI / Lightning-AI/pytorch-lightning
Graceful stop on an extrenal file marker
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Sometimes the training settings are hard to set properly, for example the `patience` parameter is set to the big value, and/or `min_delta` parameter is too small.
So I understand that the training should be stopped but there is no way for me to gracefully stop it myself.
### Pitch
Would nice to have an option to stop training with some kind of signal, the easiest way is just to create some file with `touch stop.txt`
```python
class ForceStopCallback(pl.Callback):
"""Gracefully stops training when a flag file appears on disk.
Checks for the flag file at the end of each validation epoch and sets
``trainer.should_stop = True`` if the file exists, allowing the current
epoch to finish cleanly before stopping.
"""
def __init__(self, flag_file_path: pathlib.Path):
self.flag_file_path = flag_file_path
def on_validation_end(self, trainer: pl.Trainer, module: pl.LightningModule) -> None:
if self.flag_file_path.is_file():
msg: str = f'stopped by the explicit request from the user: {self.flag_file_path}'
lightning.pytorch.utilities.rank_zero_warn(msg, category=RuntimeWarning)
trainer.should_stop = True
```
and later
```python
self.force_stopping_callback: t.Final[ForceStopCallback] = ForceStopCallback(
flag_file_path=self.snapshot_folder_path / 'stop.txt',
)
```
### Alternatives
_No response_
### Additional context
_No response_
cc @lantiga
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the callback hook around on_validation_end and the handling of trainer.should_stop, then inspect how snapshot_folder_path is used for the stop.txt marker. Done means an external flag file can request a clean stop after validation, with the proposed warning behavior covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100