deepspeedai / deepspeedai/DeepSpeed
mpi_discovery fails on systems with hostname executable from GNU Inetutils
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hello! I'm trying to launch DeepSpeed on my local machine with Arch Linux and inetutils 1.9.4 installed, but getting the following error:
Traceback (most recent call last):
File "/home/me/projectsds/ai/venv/lib/python3.8/site-packages/deepspeed/__init__.py", line 110, in initialize
engine = DeepSpeedEngine(args=args,
File "/home/me/projectsds/ai/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 137, in __init__
init_distributed(dist_backend=self.dist_backend)
File "/home/me/projectsds/ai/venv/lib/python3.8/site-packages/deepspeed/utils/distributed.py", line 42, in init_distributed
mpi_discovery(distributed_port=distributed_port, verbose=verbose)
File "/home/me/projectsds/ai/venv/lib/python3.8/site-packages/deepspeed/utils/distributed.py", line 67, in mpi_discovery
result = subprocess.check_output(hostname_cmd, shell=True)
File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['hostname -I']' returned non-zero exit status 64.
It happens because hostname executable from Inetutils doesn't support the -I flag, but mpi_discovery function uses it.
https://github.com/microsoft/DeepSpeed/blob/8067efa41cd3a6fa67d1a7495ccceb27682bc7e5/deepspeed/utils/distributed.py#L66-L68
https://www.gnu.org/software/inetutils/manual/html_node/hostname-invocation.html
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in deepspeed/utils/distributed.py at mpi_discovery, especially the hostname_cmd invocation linked in the issue. Check how hostname output is used during MPI discovery and verify the behavior on systems where GNU Inetutils does not support -I; done means discovery no longer fails in that environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- arch-linux, python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100