Lightning-AI / Lightning-AI/pytorch-lightning

Change in logging frequency for automated logging

Open
#16,821 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

Bug description

Utilising Automated Logging with self.log and self.log_dict as described in the documentation results in a shift of the logging frequency after various amounts of steps.

change_log_freq

This is also observed in all train metrics but only in the _step. It could be mutually exclusive to the WandbLogger but has already been observed to some extent in the TensorboardLogger as reported in #13525 and more generally in #10436.

How to reproduce the bug

Call self.log(, on_step=True, on_epoch=True) in training_step and let it run for more than 15k steps (in my case).
The logging rate initial is equal to log_every_n_steps=50 for some iterations but jumps wildly around for others.

batch_size=10 (specified in self.log(batch_size=10))
test_subjects=20
samples_per_subject=10

This equals to 200 samples per epoch and 20 steps per epoch. Even if log_every_n_steps=50, this should log then precisely every 100 steps (as the first 50 is not met according to my understanding) and not jump around from 2 to 200.

Environment
  • Lightning Component: Trainer/ LightningModule
  • Python 3.9
  • Pytorch-lightning 1.9.1 (installed with pip)
  • PyTorch 1.13.1
  • CUDA/ cuDNN: cuda11.6-cudnn8
  • OS: Linux (Kernel 3.10.0-1160)
  • Running environment: server

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Trainer/LightningModule automatic logging path around self.log, self.log_dict, training_step, and log_every_n_steps, then compare the WandbLogger and TensorboardLogger behavior described in the report. Reproduce the issue beyond 15k steps using the stated batch and epoch settings; done means _step logging remains at the expected frequency without irregular jumps.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.