Lightning-AI / Lightning-AI/pytorch-lightning
Allow dynamic setup in `setup()`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
Expose dataloaders to the `LightningModule`'s `setup` method.
### Motivation
This will allow for a real dynamic setup, meaning that some layers' size can be set up correctly thanks to access to the data. One does not know how data is coming from the trainer (via datamodule or dataloaders, or even from the model itself), so this must be done in setup in the most general way.
### Pitch
```python
def setup(self, stage: str = None) -> None:
"""Called at the beginning of fit (train + validate), validate, test, or predict. This is a good hook when you need to build models dynamically or adjust something about them. This hook is called on every process when using DDP.
Args:
stage (Optional[str]): either 'fit', 'validate', 'test', or 'predict'
"""
if stage == "fit" and self.model.num_nodes_pmf is None:
loaders: DataLoader = self.trainer.train_dataloader.loaders # not possible at the moment
# get cool info from the data: batch size, statistics, etc
```
### Alternatives
At the moment, this is only possible via hooks:
```python
def on_train_start(self) -> None:
"""Called at the beginning of training after sanity check."""
if self.model.num_nodes_pmf is None:
loaders: DataLoader = self.trainer.train_dataloader.loaders # succeeds
# do stuff
```
______________________________________________________________________
#### If you enjoy Lightning, check out our other projects! ⚡
- [**Metrics**](https://github.com/Lightning-AI/metrics): Machine learning metrics for distributed, scalable PyTorch applications.
- [**Lite**](https://pytorch-lightning.readthedocs.io/en/latest/starter/lightning_lite.html): enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.
- [**Flash**](https://github.com/Lightning-AI/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.
- [**Bolts**](https://github.com/Lightning-AI/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.
- [**Lightning Transformers**](https://github.com/Lightning-AI/lightning-transformers): Flexible interface for high-performance research using SOTA Transformers leveraging PyTorch Lightning, Transformers, and Hydra.
cc @borda
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing LightningModule.setup and the existing on_train_start hook, especially access through self.trainer.train_dataloader.loaders. Compare the lifecycle timing and determine how setup should expose dataloaders across the documented fit, validate, test, and predict stages; done means the requested access works without relying on the later hook.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100