Lightning-AI / Lightning-AI/pytorch-lightning
Wandb checkpoints are not logged in experiment's directory
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
Bug description
When using the Wandb Logger and setting the log_model=True, the model checkpoint isn't saved in the wandb experiment directory, but in the separate lightning logs directory.
Current behavior:
.
├── wandb
│ ├── run-20230517_120103-pgl4q0zq
│ │ ├── files
│ │ │ ├── config.yaml
│ │ │ ├── output.log
│ │ │ ├── ...
│ │ │ └── wandb-summary.json
│ │ ├── ...
│ │ └── logs
├── lightning_logs
| └── pgl4q0zq
| └── checkpoints
| └── epoch=0-step=0.ckpt
Desired behavior:
.
├── wandb
│ ├── run-20230517_120103-pgl4q0zq
│ │ ├── files
│ │ │ ├── config.yaml
│ │ │ ├── output.log
│ │ │ ├── ...
│ │ │ ├── checkpoints
│ │ │ │ └── epoch=0-step=0.ckpt
│ │ │ └── wandb-summary.json
│ │ ├── ...
│ │ └── logs
What version are you seeing the problem on?
master
How to reproduce the bug
from lightning import LightningModule, Trainer
from lightning.pytorch.loggers import WandbLogger
from torch.utils.data import Dataset, DataLoader
import torch
class MyDataModule(Dataset):
def __getitem__(self, index) -> torch.Tensor:
return torch.rand(1)
def __len__(self) -> int:
return 5
class MyLightningModule(LightningModule):
def __init__(self):
super().__init__()
self.model = torch.nn.Linear(1, 1)
def forward(self, x):
return self.model(x)
def training_step(self, batch, batch_idx):
return self.model(batch).mean()
def configure_optimizers(self):
return torch.optim.Adam(self.parameters())
model = MyLightningModule()
data = MyDataModule()
loader = DataLoader(data, batch_size=1)
trainer = Trainer(logger=WandbLogger(log_model=True))
trainer.fit(model, loader)
Error messages and logs
# Error messages and logs here please
Environment
Current environment
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
More info
I see two problems which need to be addressed:
-
The WandbLogger doesn't use the experiment's files directory as its
save_dirproperty
This can can fixed by replacing this line) byself.experiment.dir -
The ModelCheckpoint adds the name and version of the logger to the checkpoint path when resolving the checkpoint directory. The culprit is the
__resolve_ckpt_dirfunction. I would propose moving the name and version of the logger as well as the save_dir function to the actual logger, as each logger may have a different strategy for integrating the version and name. For example, wandb does this by default and creates the files directory for saving artifacts.
cc @awaelchli @morganmcg1 @borisdayma @scottire @parambharat
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the layout with WandbLogger(log_model=True), then inspect src/lightning/pytorch/loggers/wandb.py at the save_dir handling and src/lightning/pytorch/callbacks/model_checkpoint.py at __resolve_ckpt_dir. Trace how the logger experiment directory and checkpoint path are resolved. Done means checkpoints are stored under the WandB run's files directory without breaking checkpoint path behavior for other loggers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100