Lightning-AI / Lightning-AI/pytorch-lightning

Allow dynamic setup in `setup()`

Open
#15,114 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discussion feature
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature

Expose dataloaders to the `LightningModule`'s `setup` method.

### Motivation

This will allow for a real dynamic setup, meaning that some layers' size can be set up correctly thanks to access to the data. One does not know how data is coming from the trainer (via datamodule or dataloaders, or even from the model itself), so this must be done in setup in the most general way.

### Pitch

```python
def setup(self, stage: str = None) -> None:
"""Called at the beginning of fit (train + validate), validate, test, or predict. This is a good hook when you need to build models dynamically or adjust something about them. This hook is called on every process when using DDP.
Args:
stage (Optional[str]): either 'fit', 'validate', 'test', or 'predict'
"""
if stage == "fit" and self.model.num_nodes_pmf is None:
loaders: DataLoader = self.trainer.train_dataloader.loaders # not possible at the moment
# get cool info from the data: batch size, statistics, etc
```

### Alternatives

At the moment, this is only possible via hooks:

```python
def on_train_start(self) -> None:
"""Called at the beginning of training after sanity check."""
if self.model.num_nodes_pmf is None:
loaders: DataLoader = self.trainer.train_dataloader.loaders # succeeds
# do stuff
```

______________________________________________________________________

#### If you enjoy Lightning, check out our other projects! ⚡

- [**Metrics**](https://github.com/Lightning-AI/metrics): Machine learning metrics for distributed, scalable PyTorch applications.

- [**Lite**](https://pytorch-lightning.readthedocs.io/en/latest/starter/lightning_lite.html): enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

- [**Flash**](https://github.com/Lightning-AI/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

- [**Bolts**](https://github.com/Lightning-AI/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

- [**Lightning Transformers**](https://github.com/Lightning-AI/lightning-transformers): Flexible interface for high-performance research using SOTA Transformers leveraging PyTorch Lightning, Transformers, and Hydra.

cc @borda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing LightningModule.setup and the existing on_train_start hook, especially access through self.trainer.train_dataloader.loaders. Compare the lifecycle timing and determine how setup should expose dataloaders across the documented fit, validate, test, and predict stages; done means the requested access works without relying on the later hook.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.