Lightning-AI / Lightning-AI/pytorch-lightning

EarlyStopping override disrupts wandb logging frequency

Open
#19,990 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug callback: early stopping logger: wandb
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

When an `EarlyStopping` callback would halt the training before `min_epochs` has elapsed, `EarlyStopping` is (correctly) overridden, and prints the warning message given below. However, at the exact step number when the warning was printed, `WandbLogger` suddenly begins logging the train metrics for every single batch. This results in slowed training and strange output graphs.

### What version are you seeing the problem on?

v2.2

### How to reproduce the bug

```python
import torch
from lightning.pytorch import LightningModule, Trainer, seed_everything
from torch.utils.data import DataLoader, Dataset
from lightning.pytorch.loggers import WandbLogger
from lightning.pytorch.callbacks import EarlyStopping

class RandomDataset(Dataset):
def __init__(self, size, length):
self.len = length
self.data = torch.randn(length, size)

def __getitem__(self, index):
return self.data[index]

def __len__(self):
return self.len

class BoringModel(LightningModule):
def __init__(self):
super().__init__()
self.layer = torch.nn.Linear(32, 2)

def forward(self, x):
return self.layer(x)

def training_step(self, batch, batch_idx):
loss = self(batch).sum()
self.log("train_loss", loss)
return {"loss": loss}

def validation_step(self, batch, batch_idx):
loss = self(batch).sum()
self.log("valid_loss", loss)

def test_step(self, batch, batch_idx):
loss = self(batch).sum()
self.log("test_loss", loss)

def configure_optimizers(self):
return torch.optim.SGD(self.layer.parameters(), lr=0.1)

def run():
seed_everything(42, workers=True)
wandb_logger = WandbLogger(project="bug-report",
entity="example-user", name="debug_logging")

early_stopping_callback = EarlyStopping(monitor="train_loss", patience=2)

callbacks = [early_stopping_callback]

kwargs = {

"log_every_n_steps": 8,
"logger": wandb_logger,
"num_sanity_val_steps": 0,
"callbacks": callbacks,
"val_check_interval": 0.1,
"max_epochs": 10,
"min_epochs": 2,
"deterministic": True

}

train_data = DataLoader(RandomDataset(32, 64), batch_size=2)
val_data = DataLoader(RandomDataset(32, 64), batch_size=2)

model = BoringModel()
trainer = Trainer(**kwargs)
trainer.fit(model, train_dataloaders=train_data, val_dataloaders=val_data)

if __name__ == "__main__":
run()
```

### Error messages and logs

```
Epoch 0: 38%|████████████████████████████████████████▉ | 12/32 [00:00<00:00, 53.14it/s, v_num=yqz5]

Trainer was signaled to stop but the required `min_epochs=2` or `min_steps=None` has not been met. Training will continue...
```

### Environment

Current environment

* CUDA:
- GPU: None
- available: False
- version: None
* Lightning:
- lightning: 2.2.5
- lightning-utilities: 0.11.2
- pytorch-lightning: 2.2.2
- torch: 2.3.0
- torchmetrics: 1.4.0.post0
* Packages:
- appdirs: 1.4.4
- appnope: 0.1.4
- asttokens: 2.4.1
- brotli: 1.1.0
- certifi: 2024.6.2
- chardet: 5.2.0
- charset-normalizer: 3.3.2
- click: 8.1.7
- colorama: 0.4.6
- comm: 0.2.2
- contourpy: 1.2.1
- cycler: 0.12.1
- debugpy: 1.8.1
- decorator: 5.1.1
- docker-pycreds: 0.4.0
- exceptiongroup: 1.2.0
- executing: 2.0.1
- filelock: 3.14.0
- fonttools: 4.53.0
- freetype-py: 2.3.0
- fsspec: 2024.6.0
- gitdb: 4.0.11
- gitpython: 3.1.43
- gmpy2: 2.1.5
- greenlet: 3.0.3
- idna: 3.7
- importlib-metadata: 7.1.0
- ipykernel: 6.29.3
- ipython: 8.25.0
- jedi: 0.19.1
- jinja2: 3.1.4
- joblib: 1.4.2
- jupyter-client: 8.6.2
- jupyter-core: 5.7.2
- kiwisolver: 1.4.5
- lightning: 2.2.5
- lightning-utilities: 0.11.2
- markupsafe: 2.1.5
- matplotlib: 3.8.4
- matplotlib-inline: 0.1.7
- mpmath: 1.3.0
- munkres: 1.1.4
- nest-asyncio: 1.6.0
- networkx: 3.3
- numexpr: 2.10.0
- numpy: 1.26.4
- packaging: 24.0
- pandas: 2.2.2
- parso: 0.8.4
- pathtools: 0.1.2
- pexpect: 4.9.0
- pickleshare: 0.7.5
- pillow: 10.3.0
- pip: 24.0
- platformdirs: 4.2.2
- prompt-toolkit: 3.0.46
- protobuf: 4.25.3
- psutil: 5.9.8
- ptyprocess: 0.7.0
- pure-eval: 0.2.2
- py-cpuinfo: 9.0.0
- pycairo: 1.26.0
- pygments: 2.18.0
- pyparsing: 3.1.2
- pysocks: 1.7.1
- python-dateutil: 2.9.0
- pytorch-lightning: 2.2.2
- pytz: 2024.1
- pyyaml: 6.0.1
- pyzmq: 26.0.3
- rdkit: 2024.3.3
- reportlab: 4.1.0
- requests: 2.32.3
- rlpycairo: 0.2.0
- scikit-learn: 1.5.0
- scipy: 1.13.1
- sentry-sdk: 2.4.0
- setproctitle: 1.3.3
- setuptools: 70.0.0
- six: 1.16.0
- smmap: 5.0.0
- sqlalchemy: 2.0.30
- stack-data: 0.6.2
- sympy: 1.12
- tables: 3.9.2
- threadpoolctl: 3.5.0
- torch: 2.3.0
- torchmetrics: 1.4.0.post0
- tornado: 6.4.1
- tqdm: 4.66.4
- traitlets: 5.14.3
- typing-extensions: 4.12.1
- tzdata: 2024.1
- urllib3: 2.2.1
- wandb: 0.16.5
- wcwidth: 0.2.13
- wheel: 0.43.0
- zipp: 3.17.0
* System:
- OS: Darwin
- architecture:
- 64bit
-
- processor: arm
- python: 3.11.9
- release: 23.5.0
- version: Darwin Kernel Version 23.5.0: Wed May 1 20:19:05 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8112

### More info

The symptoms of this bug are somewhat similar to those of #16821 and #13525, but based on those threads it seems like the causes may be different.

cc @lantiga @morganmcg1 @borisdayma @scottire @parambharat

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Python reproducer with EarlyStopping, WandbLogger, and log_every_n_steps=8. Trace the interaction when the min_epochs warning is emitted, focusing on why train metrics switch to per-batch logging. Done means logging remains at the configured frequency after EarlyStopping is overridden, with a regression test covering the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.