Lightning-AI / Lightning-AI/pytorch-lightning
Support relative paths in ModelCheckpoint state
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
It would be great if `ModelCheckpoint` internal state supported relative paths.
### Motivation
Currently, if you specify a relative path for `dirpath`, `ModelCheckpoint` converts it to an absolute path under the hood. This makes it hard to resume training if the log directory is moved, or if resuming training from a different server with a different directory structure.
For example, I specify the relative path `a/b/c`, and `ModelCheckpoint` converts it to `cwd()/a/b/c`. The model trains correctly for a while. Then, HTCondor reschedules my job on a different server. Now, the `cwd()` is different, even though the relative path `a/b/c` is still the same. The job is unable to resume from checkpoint and I get:
> UserWarning: The dirpath has changed from to , therefore `best_model_score`, `kth_best_model_path`, `kth_value`, `last_model_path` and `best_k_models` won’t be reloaded.
### Pitch
Create an argument `relative_paths=True` that would allow `ModelCheckpoint` to use relative paths in its internal state.
### Alternatives
User can create their own checkpoint callback that supports relative paths. But, it would be much nicer if Lightning supported it :-)
### Additional context
______________________________________________________________________
#### If you enjoy Lightning, check out our other projects! ⚡
- [**Metrics**](https://github.com/Lightning-AI/metrics): Machine learning metrics for distributed, scalable PyTorch applications.
- [**Lite**](https://pytorch-lightning.readthedocs.io/en/latest/starter/lightning_lite.html): enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.
- [**Flash**](https://github.com/Lightning-AI/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.
- [**Bolts**](https://github.com/Lightning-AI/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.
- [**Lightning Transformers**](https://github.com/Lightning-AI/lightning-transformers): Flexible interface for high-performance research using SOTA Transformers leveraging PyTorch Lightning, Transformers, and Hydra.
cc @borda @carmocca @awaelchli @ninginthecloud @jjenniferdai @rohitgr7 @akihironitta
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Python ModelCheckpoint callback and the tests covering checkpoint state restoration. Reproduce a resume with a relative dirpath after changing the working directory, then verify that the internal paths and saved state restore correctly without the directory-change warning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100