Lightning-AI / Lightning-AI/pytorch-lightning

Using the WANDB logger ignores ModelCheckpoints.

Open
#17,147 3 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug logger: wandb
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description
UPDATE: it seems that the problem only occurs when 'save_last' is set to True.
to my understanding, this should not be an issue, as this flag does not cause any additional checkpoints to be created!

When using the WandB logger, and and ModelCheckpoints with some condition, the WandB logger saves checkpoints every epoch (even if the ModelCheckpoints does not indicate that it has to save a checkpoint).
This is problematic for training large models where checkpoints are large and need time and space to be made!
note: I have tried this on multiple machines, with the same version of PL and WANDB mentioned below.

### How to reproduce the bug

```python
wandb_logger = WandbLogger(log_model="all")
checkpoint_callback = ModelCheckpoint(monitor="val_accuracy", mode="max", save_last=True)
trainer = Trainer(logger=wandb_logger, callbacks=[checkpoint_callback])
```

### Error messages and logs

_No response_

### Environment

Current environment

```
* CUDA:
- version: 11.6
* Lightning:
- lightning-utilities: 0.7.1
- pytorch-lightning: 1.9.4
- torch: 1.13.1
* WANDB:
* - wandb: 1.9.
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.7.16
- version: #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023
```

### More info

_No response_

cc @awaelchli @morganmcg1 @borisdayma @scottire @parambharat

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the WandbLogger and ModelCheckpoint entry points and reproduce the provided Trainer configuration, especially with log_model="all" and save_last=True. Trace why checkpoint logging occurs every epoch despite the monitor condition. Done means only the checkpoints required by ModelCheckpoint are sent to WandB, with regression coverage for the reported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.