Lightning-AI / Lightning-AI/pytorch-lightning

`self.log` raised error when number of dataloader is not consistent

Open
#16,431 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted logging
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

Hi all,

I posted a discussion in Lightning.ai forum [here](https://lightning.ai/forums/t/self-log-raised-error-when-number-of-dataloader-is-not-consistent/2167) and @awaelchli suggested me reporting an issue.

There might be a bug in the way `self.log` recording `dataloader_idx`. If we have two validation dataloaders, says A and B. We use A every epoch, but only use B every 2 epoch. However, while using `self.log`, in `validation_step()`, an error would show up:

```
You called self.log({name}, ...) twice in {fx} with different arguments. This is not allowed
```
(see [here](https://github.com/Lightning-AI/lightning/blob/15ef52bc732d1f907de4de58683f131652c0d68c/src/pytorch_lightning/trainer/connectors/logger_connector/result.py#L489))

The way I implement it was by switching the available dataloaders in `val_dataloader()` and reload dataloader every epoch.
```python
def val_dataloader():
if self.should_run_B():
return [loader_A, loader_B]
else:
return [loader_A]
```

I notice that when we are at the epoch that only use one validation dataloader, dataloader_idx is always None. On the other hand, when we have two validation dataloaders, dataloader_idx would be sequentially presented as 0 or 1 in validation_step. If I understand correctly, this is the main reason causing the error.

Another interesting finding is that if we set `add_dataloader_idx=True` for all `self.log`, the program would run without error. But the tensorboard logging would be wrong that shows and `c_0` and`c_0/dataloader_idx_0` together (See snapshot below). These two were meant to remain in the same figure but somehow it got split into two figures. Probably it is because of the alternating dataloader design.
image

### How to reproduce the bug

```python
import torch
from torch.utils.data import DataLoader
from pytorch_lightning.demos.boring_classes import BoringModel, RandomDataset
from pytorch_lightning import Trainer

class TestModel(BoringModel):
def training_step(self, batch, batch_idx):
out = super().training_step(batch, batch_idx)
self.log("a", out["loss"])
self.log("b", out["loss"], on_step=True, on_epoch=True)
return out

def validation_step(self, batch, batch_idx, dataloader_idx=0):
out = super().validation_step(batch, batch_idx)
if dataloader_idx == 0:
self.log("c_0", out["x"], add_dataloader_idx=False)
self.log("d_0", out["x"], on_step=True, on_epoch=True, add_dataloader_idx=False)
elif dataloader_idx == 1:
self.log("c_1", out["x"], add_dataloader_idx=False)
self.log("d_1", out["x"], on_step=True, on_epoch=True, add_dataloader_idx=False)
return out

def validation_epoch_end(self, outputs):
self.log("g", torch.tensor(2, device=self.device), on_epoch=True)

def val_dataloader(self):
if self.current_epoch % 2:
return [DataLoader(RandomDataset(32, 64)), DataLoader(RandomDataset(32, 64))]
else:
return [DataLoader(RandomDataset(32, 64))]

model = TestModel()

trainer = Trainer(
reload_dataloaders_every_n_epochs=1,
default_root_dir="test",
max_epochs=10,
log_every_n_steps=1,
enable_model_summary=False,
)
trainer.fit(model)
```

### Error messages and logs

```
You called `self.log(c_0, ...)` twice in `validation_step` with different arguments. This is not allowed
```

### Environment

* CUDA:
- GPU:
- NVIDIA TITAN RTX
- available: True
- version: 11.7
* Lightning:
- pytorch-lightning: 1.7.2
- pytorch-quantization: 2.1.2
- torch: 1.12.0a0+8a1a93a
- torch-tensorrt: 1.1.0a0
- torchmetrics: 0.9.3
- torchtext: 0.13.0a0
- torchvision: 0.13.0a0
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.8.13
- version: #63-Ubuntu SMP Thu Nov 24 13:43:17 UTC 2022

### More info

_No response_

cc @carmocca @Blaizzy

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducer in the issue and inspect src/pytorch_lightning/trainer/connectors/logger_connector/result.py around the linked argument-consistency check. Run the alternating-dataloader example and compare validation_step logging across epochs, including the TensorBoard names. Done means the reproducer no longer raises the self.log error and logging remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.