Lightning-AI / Lightning-AI/pytorch-lightning

Make lazy initialization in plugins more robust

Open
#7,650 4 comments 0 reactions 1 assignee View on GitHub

@edward-io is already working on this.

Since Feb 11, 2022.

distributed feature good first issue help wanted let's do it!
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature

We currently set some attributes like sync_batch_norm, num_nodes, devices etc. lazily in the accelerator connector, so the user does not have to provide them at instantiation.

Example:

```python
Trainer(gpus=4, plugins=DDPPlugin(find_unused_parameters=True)) # plugin may require gpus, num_nodes etc.

# AcceleratorConnector does this:
training_type_plugin.num_nodes = ...
```

This is fragile. Some attributes may have to be recomputed based on the order in which others are set.

### Pitch

Provide one single lazy init method that takes all arguments required. The plugin is responsible for making sure dependencies are resolved in one place:

```python
class DDPPlugin(...):

def lazy_init(self, **kwargs):
self.num_nodes = kwargs.get("num_nodes")
self.num_processes = ...

```

With `lazy_init`, the `_configure_launcher` method (#11643) would become obsolete. It can be merged together into `lazy_init`.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.