Lightning-AI / Lightning-AI/pytorch-lightning
Using the WANDB logger ignores ModelCheckpoints.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
UPDATE: it seems that the problem only occurs when 'save_last' is set to True.
to my understanding, this should not be an issue, as this flag does not cause any additional checkpoints to be created!
When using the WandB logger, and and ModelCheckpoints with some condition, the WandB logger saves checkpoints every epoch (even if the ModelCheckpoints does not indicate that it has to save a checkpoint).
This is problematic for training large models where checkpoints are large and need time and space to be made!
note: I have tried this on multiple machines, with the same version of PL and WANDB mentioned below.
### How to reproduce the bug
```python
wandb_logger = WandbLogger(log_model="all")
checkpoint_callback = ModelCheckpoint(monitor="val_accuracy", mode="max", save_last=True)
trainer = Trainer(logger=wandb_logger, callbacks=[checkpoint_callback])
```
### Error messages and logs
_No response_
### Environment
Current environment
```
* CUDA:
- version: 11.6
* Lightning:
- lightning-utilities: 0.7.1
- pytorch-lightning: 1.9.4
- torch: 1.13.1
* WANDB:
* - wandb: 1.9.
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.7.16
- version: #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023
```
### More info
_No response_
cc @awaelchli @morganmcg1 @borisdayma @scottire @parambharat
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the WandbLogger and ModelCheckpoint entry points and reproduce the provided Trainer configuration, especially with log_model="all" and save_last=True. Trace why checkpoint logging occurs every epoch despite the monitor condition. Done means only the checkpoints required by ModelCheckpoint are sent to WandB, with regression coverage for the reported configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100