Lightning-AI / Lightning-AI/pytorch-lightning

Allow obtaining num_nodes from ClusterEnvironment

Open
#7,361 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

environment feature good first issue help wanted
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature
num_nodes must currently be specified manually by the user. However, the number of nodes is generally known in a cluster environment [1] and could be provided by and initialized from [ClusterEnvironment](https://pytorch-lightning.readthedocs.io/en/latest/clouds/cluster.html)

[1] For example $AWS_BATCH_JOB_NUM_NODES in https://docs.aws.amazon.com/batch/latest/userguide/multi-node-parallel-jobs.html

### Pitch

https://github.com/PyTorchLightning/pytorch-lightning/blob/763a9a9495977b23cbd6a57f10253b662fd592a5/pytorch_lightning/plugins/training_type/ddp.py#L62-L75

could be updated to initialize num_nodes from ClusterEnvironment if ClusterEnvironment is provided and implements a `num_nodes` method.

cc @borda @awaelchli @ananthsub

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pytorch_lightning/plugins/training_type/ddp.py at the linked num_nodes initialization, then inspect ClusterEnvironment to determine how an optional num_nodes method should be used. The change is complete when num_nodes can be initialized from the cluster environment when that method is available, while manual configuration remains supported otherwise.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.