huggingface / huggingface/accelerate
[feature request] making the use of the `accelerate` launcher optional
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.5k
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 27
Description
Sometimes there is a need to be able to launch an Accelerate-based program w/o the `accelerate` launcher - for example, in case once wants to use the super-optimized SLURM resource placements and having `srun` launch all the individual per-gpu processes.
e.g. Lightning doesn't use a launcher, e.g. see [here](https://github.com/stas00/ml-engineering/blob/master/orchestration/slurm/launchers/lightning-launcher.slurm) and here is how I can launch a normal torch.distributed multi-gpu program w/o `torchrun`: [here](https://github.com/stas00/ml-engineering/blob/master/orchestration/slurm/launchers/srun-launcher.slurm) -or more specifically it's just:
srun bash -c "RANK=\$SLURM_PROCID LOCAL_RANK=\$SLURM_LOCALID python my-dist-prog.py"
which to the program is just:
RANK=... LOCAL_RANK=... python my-dist-prog.py
Of course, we want the advanced features supported - so Deepspeed/FSDP, etc.
And ideally, it'd be the same accelerate config file - so it can be re-used w/ and w/o the launcher.
Thank you.
Contributor guide
Assessment
This issue has not been assessed yet.