pytorch / pytorch/ignite

Distributed may deadlock in `one_rank_first` when an exception occurs on the target rank

Open
#3,675 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.8k
Forks
726
Avg merge
5d 21h
Merged PRs (30d)
5

Description

Bug description

In ignite/distributed/utils.py, the one_rank_first context manager is susceptible to a distributed deadlock if an exception occurs within the context block on the target rank.

The current implementation uses two barriers to synchronize ranks. If the process with the designated rank encounters an error (e.g., a network failure during data download or a disk I/O error) within the yield block, it will never reach the second barrier. Consequently, all other processes that are either waiting at the first barrier or expecting to synchronize at the second will hang indefinitely.

Steps to reproduce

Run the following logic in a distributed environment (2+ ranks):

import ignite.distributed as idist

# Simulate a crash only on rank 0
with idist.one_rank_first(rank=0):
    if idist.get_rank() == 0:
        raise RuntimeError("Rank 0 crashed!")
    # Other ranks are now stuck at the first barrier or will hit the second

Expected behavior

The exception should propagate, and the entire distributed job should terminate gracefully. Instead, the non-crashed ranks hang, requiring a manual kill of the processes.

Code Snippet

Current implementation in ignite/distributed/utils.py:

if current_rank != rank:
    barrier()

yield  # <--- If an exception happens here on 'rank'

if current_rank == rank:
    barrier() # <--- Other ranks never reach or pass synchronization

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in ignite/distributed/utils.py and inspect the one_rank_first context manager and its two barrier calls. Run the two-rank reproduction from the issue, including the exception on the target rank, and trace which ranks remain blocked. Done means the exception propagates and the distributed job terminates without leaving other ranks hanging.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.