Lightning-AI / Lightning-AI/pytorch-lightning
LightningModule self.log add_dataloader_idx doesn't reduce properly the metric across dataloaders
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🐛 Bug
### To Reproduce
The current behavior with `add_dataloader_idx` seems quite confusing to me. As a user, I don't know if I would expect to get the value reduced across all dataloaders and be added to both results objects.
```py
def test_multiple_dataloaders_logging(tmpdir):
class TestModel(BoringModel):
def validation_step(self, batch, batch_idx, dataloader_idx):
self.log("value_1", dataloader_idx, add_dataloader_idx=False)
self.log("value_2", dataloader_idx, add_dataloader_idx=True)
def val_dataloader(self):
return [self.train_dataloader(), self.train_dataloader()]
model = TestModel()
model.validation_epoch_end = None
trainer = Trainer(default_root_dir=tmpdir)
results = trainer.validate(model)
assert results == [
{"value_2/dataloader_idx_0": 0.0, "value_1": 0.5},
{"value_2/dataloader_idx_1": 1.0, "value_1": 0.5},
]
```
### Expected behavior
### Environment
- PyTorch Lightning Version (e.g., 1.5.0):
- PyTorch Version (e.g., 1.10):
- Python version (e.g., 3.9):
- OS (e.g., Linux):
- CUDA/cuDNN version:
- GPU models and configuration:
- How you installed PyTorch (`conda`, `pip`, source):
- If compiling from source, the output of `torch.__config__.show()`:
- Any other relevant information:
### Additional context
cc @tchaton
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with LightningModule.self.log, the validation_step example, and Trainer.validate using the provided BoringModel reproduction. Trace how value_1 and value_2 are reduced across the two dataloaders, then add or update the multiple-dataloader logging test so the agreed reduction behavior is verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100