Lightning-AI / Lightning-AI/pytorch-lightning
EarlyStopping override disrupts wandb logging frequency
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
When an `EarlyStopping` callback would halt the training before `min_epochs` has elapsed, `EarlyStopping` is (correctly) overridden, and prints the warning message given below. However, at the exact step number when the warning was printed, `WandbLogger` suddenly begins logging the train metrics for every single batch. This results in slowed training and strange output graphs.
### What version are you seeing the problem on?
v2.2
### How to reproduce the bug
```python
import torch
from lightning.pytorch import LightningModule, Trainer, seed_everything
from torch.utils.data import DataLoader, Dataset
from lightning.pytorch.loggers import WandbLogger
from lightning.pytorch.callbacks import EarlyStopping
class RandomDataset(Dataset):
def __init__(self, size, length):
self.len = length
self.data = torch.randn(length, size)
def __getitem__(self, index):
return self.data[index]
def __len__(self):
return self.len
class BoringModel(LightningModule):
def __init__(self):
super().__init__()
self.layer = torch.nn.Linear(32, 2)
def forward(self, x):
return self.layer(x)
def training_step(self, batch, batch_idx):
loss = self(batch).sum()
self.log("train_loss", loss)
return {"loss": loss}
def validation_step(self, batch, batch_idx):
loss = self(batch).sum()
self.log("valid_loss", loss)
def test_step(self, batch, batch_idx):
loss = self(batch).sum()
self.log("test_loss", loss)
def configure_optimizers(self):
return torch.optim.SGD(self.layer.parameters(), lr=0.1)
def run():
seed_everything(42, workers=True)
wandb_logger = WandbLogger(project="bug-report",
entity="example-user", name="debug_logging")
early_stopping_callback = EarlyStopping(monitor="train_loss", patience=2)
callbacks = [early_stopping_callback]
kwargs = {
"log_every_n_steps": 8,
"logger": wandb_logger,
"num_sanity_val_steps": 0,
"callbacks": callbacks,
"val_check_interval": 0.1,
"max_epochs": 10,
"min_epochs": 2,
"deterministic": True
}
train_data = DataLoader(RandomDataset(32, 64), batch_size=2)
val_data = DataLoader(RandomDataset(32, 64), batch_size=2)
model = BoringModel()
trainer = Trainer(**kwargs)
trainer.fit(model, train_dataloaders=train_data, val_dataloaders=val_data)
if __name__ == "__main__":
run()
```
### Error messages and logs
```
Epoch 0: 38%|████████████████████████████████████████▉ | 12/32 [00:00<00:00, 53.14it/s, v_num=yqz5]
Trainer was signaled to stop but the required `min_epochs=2` or `min_steps=None` has not been met. Training will continue...
```
### Environment
Current environment
* CUDA:
- GPU: None
- available: False
- version: None
* Lightning:
- lightning: 2.2.5
- lightning-utilities: 0.11.2
- pytorch-lightning: 2.2.2
- torch: 2.3.0
- torchmetrics: 1.4.0.post0
* Packages:
- appdirs: 1.4.4
- appnope: 0.1.4
- asttokens: 2.4.1
- brotli: 1.1.0
- certifi: 2024.6.2
- chardet: 5.2.0
- charset-normalizer: 3.3.2
- click: 8.1.7
- colorama: 0.4.6
- comm: 0.2.2
- contourpy: 1.2.1
- cycler: 0.12.1
- debugpy: 1.8.1
- decorator: 5.1.1
- docker-pycreds: 0.4.0
- exceptiongroup: 1.2.0
- executing: 2.0.1
- filelock: 3.14.0
- fonttools: 4.53.0
- freetype-py: 2.3.0
- fsspec: 2024.6.0
- gitdb: 4.0.11
- gitpython: 3.1.43
- gmpy2: 2.1.5
- greenlet: 3.0.3
- idna: 3.7
- importlib-metadata: 7.1.0
- ipykernel: 6.29.3
- ipython: 8.25.0
- jedi: 0.19.1
- jinja2: 3.1.4
- joblib: 1.4.2
- jupyter-client: 8.6.2
- jupyter-core: 5.7.2
- kiwisolver: 1.4.5
- lightning: 2.2.5
- lightning-utilities: 0.11.2
- markupsafe: 2.1.5
- matplotlib: 3.8.4
- matplotlib-inline: 0.1.7
- mpmath: 1.3.0
- munkres: 1.1.4
- nest-asyncio: 1.6.0
- networkx: 3.3
- numexpr: 2.10.0
- numpy: 1.26.4
- packaging: 24.0
- pandas: 2.2.2
- parso: 0.8.4
- pathtools: 0.1.2
- pexpect: 4.9.0
- pickleshare: 0.7.5
- pillow: 10.3.0
- pip: 24.0
- platformdirs: 4.2.2
- prompt-toolkit: 3.0.46
- protobuf: 4.25.3
- psutil: 5.9.8
- ptyprocess: 0.7.0
- pure-eval: 0.2.2
- py-cpuinfo: 9.0.0
- pycairo: 1.26.0
- pygments: 2.18.0
- pyparsing: 3.1.2
- pysocks: 1.7.1
- python-dateutil: 2.9.0
- pytorch-lightning: 2.2.2
- pytz: 2024.1
- pyyaml: 6.0.1
- pyzmq: 26.0.3
- rdkit: 2024.3.3
- reportlab: 4.1.0
- requests: 2.32.3
- rlpycairo: 0.2.0
- scikit-learn: 1.5.0
- scipy: 1.13.1
- sentry-sdk: 2.4.0
- setproctitle: 1.3.3
- setuptools: 70.0.0
- six: 1.16.0
- smmap: 5.0.0
- sqlalchemy: 2.0.30
- stack-data: 0.6.2
- sympy: 1.12
- tables: 3.9.2
- threadpoolctl: 3.5.0
- torch: 2.3.0
- torchmetrics: 1.4.0.post0
- tornado: 6.4.1
- tqdm: 4.66.4
- traitlets: 5.14.3
- typing-extensions: 4.12.1
- tzdata: 2024.1
- urllib3: 2.2.1
- wandb: 0.16.5
- wcwidth: 0.2.13
- wheel: 0.43.0
- zipp: 3.17.0
* System:
- OS: Darwin
- architecture:
- 64bit
-
- processor: arm
- python: 3.11.9
- release: 23.5.0
- version: Darwin Kernel Version 23.5.0: Wed May 1 20:19:05 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8112
### More info
The symptoms of this bug are somewhat similar to those of #16821 and #13525, but based on those threads it seems like the causes may be different.
cc @lantiga @morganmcg1 @borisdayma @scottire @parambharat
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided Python reproducer with EarlyStopping, WandbLogger, and log_every_n_steps=8. Trace the interaction when the min_epochs warning is emitted, focusing on why train metrics switch to per-batch logging. Done means logging remains at the configured frequency after EarlyStopping is overridden, with a regression test covering the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100