deepspeedai / deepspeedai/DeepSpeed
[BUG] pdsh runner doesn't work with tqdm bar
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
Training tqdm bar won't show in the rank0 console if pdsh is used as the launcher.
To Reproduce
Run any multi-node training on an example script, the tqdm bar from local rank (localhost) won't show up but show as a blank line.
I assume the root cause it pdsh tries to ssh into even localhost, then the bar cannot be logged through a ssh session somehow.
Expected behavior
Be able to see the tqdm bar.
System info (please complete the following information):
- OS: Ubuntu 22.04
- x8 GPUs each node
- Python version 3.12.8
- 4 Nodes, same network
Launcher context
PDSH
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the PDSH launcher using a multi-node example script and compare the rank0 console with another launcher. Trace how the local-rank tqdm output is handled when PDSH connects to localhost; done means the tqdm bar is visible in the rank0 console during multi-node training.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100