Lightning-AI / Lightning-AI/pytorch-lightning

Wandb checkpoints are not logged in experiment's directory

Open
#17,654 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug logger: wandb ver: 2.1.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

Bug description

When using the Wandb Logger and setting the log_model=True, the model checkpoint isn't saved in the wandb experiment directory, but in the separate lightning logs directory.

Current behavior:

.
├── wandb
│   ├── run-20230517_120103-pgl4q0zq
│   │   ├── files
│   │   │   ├── config.yaml
│   │   │   ├── output.log
│   │   │   ├── ...
│   │   │   └── wandb-summary.json
│   │   ├── ...
│   │   └── logs
├── lightning_logs
|   └── pgl4q0zq
|       └── checkpoints
|           └── epoch=0-step=0.ckpt

Desired behavior:

.
├── wandb
│   ├── run-20230517_120103-pgl4q0zq
│   │   ├── files
│   │   │   ├── config.yaml
│   │   │   ├── output.log
│   │   │   ├── ...
│   │   │   ├── checkpoints
│   │   │   │   └── epoch=0-step=0.ckpt
│   │   │   └── wandb-summary.json
│   │   ├── ...
│   │   └── logs
What version are you seeing the problem on?

master

How to reproduce the bug
from lightning import LightningModule, Trainer
from lightning.pytorch.loggers import WandbLogger
from torch.utils.data import Dataset, DataLoader
import torch


class MyDataModule(Dataset):
    def __getitem__(self, index) -> torch.Tensor:
        return torch.rand(1)

    def __len__(self) -> int:
        return 5


class MyLightningModule(LightningModule):
    def __init__(self):
        super().__init__()
        self.model = torch.nn.Linear(1, 1)

    def forward(self, x):
        return self.model(x)

    def training_step(self, batch, batch_idx):
        return self.model(batch).mean()

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters())


model = MyLightningModule()
data = MyDataModule()
loader = DataLoader(data, batch_size=1)
trainer = Trainer(logger=WandbLogger(log_model=True))
trainer.fit(model, loader)
Error messages and logs
# Error messages and logs here please
Environment
Current environment
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
More info

I see two problems which need to be addressed:

  1. The WandbLogger doesn't use the experiment's files directory as its save_dir property
    This can can fixed by replacing this line) by self.experiment.dir

  2. The ModelCheckpoint adds the name and version of the logger to the checkpoint path when resolving the checkpoint directory. The culprit is the __resolve_ckpt_dir function. I would propose moving the name and version of the logger as well as the save_dir function to the actual logger, as each logger may have a different strategy for integrating the version and name. For example, wandb does this by default and creates the files directory for saving artifacts.

cc @awaelchli @morganmcg1 @borisdayma @scottire @parambharat

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the layout with WandbLogger(log_model=True), then inspect src/lightning/pytorch/loggers/wandb.py at the save_dir handling and src/lightning/pytorch/callbacks/model_checkpoint.py at __resolve_ckpt_dir. Trace how the logger experiment directory and checkpoint path are resolved. Done means checkpoints are stored under the WandB run's files directory without breaking checkpoint path behavior for other loggers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.