Lightning-AI / Lightning-AI/pytorch-lightning
WandbLogger marks crashed runs as "Finished" instead of "Failed" in W&B sweeps
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
When using W&B sweeps, a run that crashes via `sys.exit(0)` — or any other mechanism that causes Python to exit with code 0 — is incorrectly marked as **"Finished"** (green) in the W&B UI instead of **"Failed"**. This silently corrupts sweep results because a crashed run appears to have completed successfully.
`WandbLogger.finalize("failed")` previously did nothing (early return). Python's atexit handler then calls `wandb.finish()` with the process exit code, and since `sys.exit(0)` produces exit code 0, W&B interprets it as success.
**Expected:** The run is marked as "Failed".
**Actual:** The run is marked as "Finished".
I'll have a PR in a moment.
### What version are you seeing the problem on?
master
### Reproduced in studio
_No response_
### How to reproduce the bug
```python
Save the following as `crash_test.py`:
import sys
import torch
from lightning.pytorch import LightningModule, Trainer
from lightning.pytorch.loggers import WandbLogger
from torch.utils.data import DataLoader, TensorDataset
class CrashingModel(LightningModule):
def __init__(self):
super().__init__()
self.layer = torch.nn.Linear(1, 1)
def training_step(self, batch, batch_idx):
x, y = batch
loss = torch.nn.functional.mse_loss(self.layer(x), y)
self.log("train/loss", loss)
if batch_idx >= 2:
sys.exit(0)
return loss
def configure_optimizers(self):
return torch.optim.Adam(self.parameters())
trainer = Trainer(
max_epochs=3,
logger=WandbLogger(project="my_project"),
)
trainer.fit(CrashingModel(), DataLoader(TensorDataset(torch.randn(100, 1), torch.randn(100, 1))))
Save the following as `crash_test_sweep.yaml`:
program: crash_test.py
project: my_project
method: random
run_cap: 1
parameters:
learning_rate:
values: [0.001, 0.01, 0.1]
Then run:
wandb sweep crash_test_sweep.yaml
wandb agent
Observe that the run in the W&B sweep dashboard is marked as "Finished" (green) instead of "Failed".
```
### Error messages and logs
```
# Error messages and logs here please
```
### Environment
Current environment
* CUDA:
- GPU:
- NVIDIA GeForce RTX 3090
- NVIDIA GeForce RTX 3050
- available: True
- version: 12.8
* Lightning:
- lightning: 2.6.2
- lightning-utilities: 0.15.3
- pytorch-lightning: 2.6.2
- torch: 2.9.1
- torchmetrics: 1.8.2
- torchvision: 0.24.1
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.10.19
- release: 6.6.114.1-microsoft-standard-WSL2
### More info
_No response_
cc @ethanwharris @lantiga @morganmcg1 @borisdayma @scottire @parambharat
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing WandbLogger.finalize("failed") and the atexit path that calls wandb.finish(), then run the provided crash_test.py with crash_test_sweep.yaml through a W&B sweep. Done means a run ending through sys.exit(0) is shown as "Failed" rather than "Finished" in the sweep dashboard.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100