deepspeedai / deepspeedai/DeepSpeed

mpi_discovery fails on systems with hostname executable from GNU Inetutils

Open
#826 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Hello! I'm trying to launch DeepSpeed on my local machine with Arch Linux and inetutils 1.9.4 installed, but getting the following error:

Traceback (most recent call last):
  File "/home/me/projectsds/ai/venv/lib/python3.8/site-packages/deepspeed/__init__.py", line 110, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/home/me/projectsds/ai/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 137, in __init__
    init_distributed(dist_backend=self.dist_backend)
  File "/home/me/projectsds/ai/venv/lib/python3.8/site-packages/deepspeed/utils/distributed.py", line 42, in init_distributed
    mpi_discovery(distributed_port=distributed_port, verbose=verbose)
  File "/home/me/projectsds/ai/venv/lib/python3.8/site-packages/deepspeed/utils/distributed.py", line 67, in mpi_discovery
    result = subprocess.check_output(hostname_cmd, shell=True)
  File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['hostname -I']' returned non-zero exit status 64.

It happens because hostname executable from Inetutils doesn't support the -I flag, but mpi_discovery function uses it.
https://github.com/microsoft/DeepSpeed/blob/8067efa41cd3a6fa67d1a7495ccceb27682bc7e5/deepspeed/utils/distributed.py#L66-L68
https://www.gnu.org/software/inetutils/manual/html_node/hostname-invocation.html

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in deepspeed/utils/distributed.py at mpi_discovery, especially the hostname_cmd invocation linked in the issue. Check how hostname output is used during MPI discovery and verify the behavior on systems where GNU Inetutils does not support -I; done means discovery no longer fails in that environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
arch-linux, python
Domain
distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.