Distributed may deadlock in `one_rank_first` when an exception occurs on the target rank
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 726
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 5
Description
Bug description
In ignite/distributed/utils.py, the one_rank_first context manager is susceptible to a distributed deadlock if an exception occurs within the context block on the target rank.
The current implementation uses two barriers to synchronize ranks. If the process with the designated rank encounters an error (e.g., a network failure during data download or a disk I/O error) within the yield block, it will never reach the second barrier. Consequently, all other processes that are either waiting at the first barrier or expecting to synchronize at the second will hang indefinitely.
Steps to reproduce
Run the following logic in a distributed environment (2+ ranks):
import ignite.distributed as idist
# Simulate a crash only on rank 0
with idist.one_rank_first(rank=0):
if idist.get_rank() == 0:
raise RuntimeError("Rank 0 crashed!")
# Other ranks are now stuck at the first barrier or will hit the second
Expected behavior
The exception should propagate, and the entire distributed job should terminate gracefully. Instead, the non-crashed ranks hang, requiring a manual kill of the processes.
Code Snippet
Current implementation in ignite/distributed/utils.py:
if current_rank != rank:
barrier()
yield # <--- If an exception happens here on 'rank'
if current_rank == rank:
barrier() # <--- Other ranks never reach or pass synchronization
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in ignite/distributed/utils.py and inspect the one_rank_first context manager and its two barrier calls. Run the two-rank reproduction from the issue, including the exception on the target rank, and trace which ranks remain blocked. Done means the exception propagates and the distributed job terminates without leaving other ranks hanging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100