Lightning-AI / Lightning-AI/pytorch-lightning

LightningModule self.log add_dataloader_idx doesn't reduce properly the metric across dataloaders

Open
#11,126 14 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug priority: 1
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🐛 Bug

### To Reproduce

The current behavior with `add_dataloader_idx` seems quite confusing to me. As a user, I don't know if I would expect to get the value reduced across all dataloaders and be added to both results objects.

```py
def test_multiple_dataloaders_logging(tmpdir):
class TestModel(BoringModel):
def validation_step(self, batch, batch_idx, dataloader_idx):
self.log("value_1", dataloader_idx, add_dataloader_idx=False)
self.log("value_2", dataloader_idx, add_dataloader_idx=True)

def val_dataloader(self):
return [self.train_dataloader(), self.train_dataloader()]

model = TestModel()
model.validation_epoch_end = None
trainer = Trainer(default_root_dir=tmpdir)
results = trainer.validate(model)
assert results == [
{"value_2/dataloader_idx_0": 0.0, "value_1": 0.5},
{"value_2/dataloader_idx_1": 1.0, "value_1": 0.5},
]
```

### Expected behavior

### Environment

- PyTorch Lightning Version (e.g., 1.5.0):
- PyTorch Version (e.g., 1.10):
- Python version (e.g., 3.9):
- OS (e.g., Linux):
- CUDA/cuDNN version:
- GPU models and configuration:
- How you installed PyTorch (`conda`, `pip`, source):
- If compiling from source, the output of `torch.__config__.show()`:
- Any other relevant information:

### Additional context

cc @tchaton

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with LightningModule.self.log, the validation_step example, and Trainer.validate using the provided BoringModel reproduction. Trace how value_1 and value_2 are reduced across the two dataloaders, then add or update the multiple-dataloader logging test so the agreed reduction behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.