huggingface / huggingface/accelerate

[feature request] making the use of the `accelerate` launcher optional

Open
#2,585 6 comments 0 reactions 0 assignees View on GitHub
enhancement feature request
Dominant language
Python
Stars
9.9k
Forks
1.5k
Avg merge
5d 2h
Merged PRs (30d)
27

Description

Sometimes there is a need to be able to launch an Accelerate-based program w/o the `accelerate` launcher - for example, in case once wants to use the super-optimized SLURM resource placements and having `srun` launch all the individual per-gpu processes.

e.g. Lightning doesn't use a launcher, e.g. see [here](https://github.com/stas00/ml-engineering/blob/master/orchestration/slurm/launchers/lightning-launcher.slurm) and here is how I can launch a normal torch.distributed multi-gpu program w/o `torchrun`: [here](https://github.com/stas00/ml-engineering/blob/master/orchestration/slurm/launchers/srun-launcher.slurm) -or more specifically it's just:

srun bash -c "RANK=\$SLURM_PROCID LOCAL_RANK=\$SLURM_LOCALID python my-dist-prog.py"

which to the program is just:

RANK=... LOCAL_RANK=... python my-dist-prog.py

Of course, we want the advanced features supported - so Deepspeed/FSDP, etc.

And ideally, it'd be the same accelerate config file - so it can be re-used w/ and w/o the launcher.

Thank you.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.