Lightning-AI / Lightning-AI/pytorch-lightning
Change in logging frequency for automated logging
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
Bug description
Utilising Automated Logging with self.log and self.log_dict as described in the documentation results in a shift of the logging frequency after various amounts of steps.

This is also observed in all train metrics but only in the _step. It could be mutually exclusive to the WandbLogger but has already been observed to some extent in the TensorboardLogger as reported in #13525 and more generally in #10436.
How to reproduce the bug
Call self.log(, on_step=True, on_epoch=True) in training_step and let it run for more than 15k steps (in my case).
The logging rate initial is equal to log_every_n_steps=50 for some iterations but jumps wildly around for others.
batch_size=10 (specified in self.log(batch_size=10))
test_subjects=20
samples_per_subject=10
This equals to 200 samples per epoch and 20 steps per epoch. Even if log_every_n_steps=50, this should log then precisely every 100 steps (as the first 50 is not met according to my understanding) and not jump around from 2 to 200.
Environment
- Lightning Component: Trainer/ LightningModule
- Python 3.9
- Pytorch-lightning 1.9.1 (installed with pip)
- PyTorch 1.13.1
- CUDA/ cuDNN: cuda11.6-cudnn8
- OS: Linux (Kernel 3.10.0-1160)
- Running environment: server
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Trainer/LightningModule automatic logging path around self.log, self.log_dict, training_step, and log_every_n_steps, then compare the WandbLogger and TensorboardLogger behavior described in the report. Reproduce the issue beyond 15k steps using the stated batch and epoch settings; done means _step logging remains at the expected frequency without irregular jumps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100