Lightning-AI / Lightning-AI/pytorch-lightning
Make lazy initialization in plugins more robust
@edward-io is already working on this.
Since Feb 11, 2022.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
We currently set some attributes like sync_batch_norm, num_nodes, devices etc. lazily in the accelerator connector, so the user does not have to provide them at instantiation.
Example:
```python
Trainer(gpus=4, plugins=DDPPlugin(find_unused_parameters=True)) # plugin may require gpus, num_nodes etc.
# AcceleratorConnector does this:
training_type_plugin.num_nodes = ...
```
This is fragile. Some attributes may have to be recomputed based on the order in which others are set.
### Pitch
Provide one single lazy init method that takes all arguments required. The plugin is responsible for making sure dependencies are resolved in one place:
```python
class DDPPlugin(...):
def lazy_init(self, **kwargs):
self.num_nodes = kwargs.get("num_nodes")
self.num_processes = ...
```
With `lazy_init`, the `_configure_launcher` method (#11643) would become obsolete. It can be merged together into `lazy_init`.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.