Lightning-AI / Lightning-AI/pytorch-lightning

WandbLogger marks crashed runs as "Finished" instead of "Failed" in W&B sweeps

Open
#21,787 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug logger: wandb ver: 2.7.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

When using W&B sweeps, a run that crashes via `sys.exit(0)` — or any other mechanism that causes Python to exit with code 0 — is incorrectly marked as **"Finished"** (green) in the W&B UI instead of **"Failed"**. This silently corrupts sweep results because a crashed run appears to have completed successfully.

`WandbLogger.finalize("failed")` previously did nothing (early return). Python's atexit handler then calls `wandb.finish()` with the process exit code, and since `sys.exit(0)` produces exit code 0, W&B interprets it as success.

**Expected:** The run is marked as "Failed".
**Actual:** The run is marked as "Finished".

I'll have a PR in a moment.

### What version are you seeing the problem on?

master

### Reproduced in studio

_No response_

### How to reproduce the bug

```python
Save the following as `crash_test.py`:

import sys
import torch
from lightning.pytorch import LightningModule, Trainer
from lightning.pytorch.loggers import WandbLogger
from torch.utils.data import DataLoader, TensorDataset

class CrashingModel(LightningModule):
def __init__(self):
super().__init__()
self.layer = torch.nn.Linear(1, 1)

def training_step(self, batch, batch_idx):
x, y = batch
loss = torch.nn.functional.mse_loss(self.layer(x), y)
self.log("train/loss", loss)
if batch_idx >= 2:
sys.exit(0)
return loss

def configure_optimizers(self):
return torch.optim.Adam(self.parameters())

trainer = Trainer(
max_epochs=3,
logger=WandbLogger(project="my_project"),
)
trainer.fit(CrashingModel(), DataLoader(TensorDataset(torch.randn(100, 1), torch.randn(100, 1))))

Save the following as `crash_test_sweep.yaml`:

program: crash_test.py
project: my_project
method: random
run_cap: 1
parameters:
learning_rate:
values: [0.001, 0.01, 0.1]

Then run:

wandb sweep crash_test_sweep.yaml
wandb agent

Observe that the run in the W&B sweep dashboard is marked as "Finished" (green) instead of "Failed".
```

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

* CUDA:
- GPU:
- NVIDIA GeForce RTX 3090
- NVIDIA GeForce RTX 3050
- available: True
- version: 12.8
* Lightning:
- lightning: 2.6.2
- lightning-utilities: 0.15.3
- pytorch-lightning: 2.6.2
- torch: 2.9.1
- torchmetrics: 1.8.2
- torchvision: 0.24.1
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.10.19
- release: 6.6.114.1-microsoft-standard-WSL2

### More info

_No response_

cc @ethanwharris @lantiga @morganmcg1 @borisdayma @scottire @parambharat

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing WandbLogger.finalize("failed") and the atexit path that calls wandb.finish(), then run the provided crash_test.py with crash_test_sweep.yaml through a W&B sweep. Done means a run ending through sys.exit(0) is shown as "Failed" rather than "Finished" in the sweep dashboard.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.