ml-explore / ml-explore/mlx

[BUG] JACCL never detects a lost peer: survivors busy-wait at 100% CPU indefinitely

Open
#4,278 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug distributed low priority
Dominant language
C++
Stars
28.5k
Forks
2.3k
Avg merge
3d 8h
Merged PRs (30d)
62

Description

What happens

When a rank stops responding, the other ranks of a JACCL group wait forever. No
timeout, no error, and not idly: each survivor spins a full core.

Four M4 Pro minis, full mesh over Thunderbolt RDMA, all four in a loop of 16 MB
all_sum inside try/except. Rank 2 suspended with SIGSTOP:

  • the other three froze at the same iteration and stayed there, silent, indefinitely
  • no exception, no abort, nothing on stderr
  • ps showed all three at 99 to 100% CPU in state Rs+, so they are busy-waiting
    rather than blocked on a descriptor
  • SIGCONT and all four resumed and completed normally, no corruption, no lost work

So a transient stall is survivable, which is good, and a death is indistinguishable
from it, which is not. Nothing ever reports that a peer is gone.

Why SIGSTOP rather than killing a rank

mlx.launch kills the surviving ranks the moment any rank exits, so it hides
whatever the survivors would have done. Suspending one rank leaves the process alive,
so the launcher stays out of the way while the peer stops servicing RDMA entirely.

Cutting the victim off at the network layer does not work as
a test: with ifconfig en2/en3/en4 down, the interfaces showing neither UP nor
RUNNING and IP ping over the point to point link at 100% loss, the four rank
collective kept running at full speed. RDMA over Thunderbolt does not go through
the IP stack once queue pairs are established. The /30 addresses only bootstrap the
side channel.

Comparison with ring

Ring had the same shape and it was addressed: #4060 made a lost peer fail the pending
tasks, and #3742 turns that into a catchable error. On the three revisions, a two
rank ring peer loss gives: v0.32.0 hangs, current main aborts with exit 134, #3742
raises a catchable RuntimeError. JACCL has none of that.

What would help

Anything that ends the wait. A timeout on the completion queue poll, or a liveness
check on the side channel, with the group reporting the loss instead of spinning.
The recovery behaviour after SIGCONT suggests the wait itself is sound and only the
unbounded part is the problem.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with JACCL's completion-queue wait and side-channel liveness path, then compare the behavior and fixes described in ring issues #4060 and #3742. Reproduce the SIGSTOP scenario with the four-rank all_sum loop. Done means a lost peer ends the wait and reports an error instead of spinning indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.