[BUG] JACCL never detects a lost peer: survivors busy-wait at 100% CPU indefinitely
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 28.5k
- Forks
- 2.3k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 62
Description
What happens
When a rank stops responding, the other ranks of a JACCL group wait forever. No
timeout, no error, and not idly: each survivor spins a full core.
Four M4 Pro minis, full mesh over Thunderbolt RDMA, all four in a loop of 16 MB
all_sum inside try/except. Rank 2 suspended with SIGSTOP:
- the other three froze at the same iteration and stayed there, silent, indefinitely
- no exception, no abort, nothing on stderr
psshowed all three at 99 to 100% CPU in stateRs+, so they are busy-waiting
rather than blocked on a descriptorSIGCONTand all four resumed and completed normally, no corruption, no lost work
So a transient stall is survivable, which is good, and a death is indistinguishable
from it, which is not. Nothing ever reports that a peer is gone.
Why SIGSTOP rather than killing a rank
mlx.launch kills the surviving ranks the moment any rank exits, so it hides
whatever the survivors would have done. Suspending one rank leaves the process alive,
so the launcher stays out of the way while the peer stops servicing RDMA entirely.
Cutting the victim off at the network layer does not work as
a test: with ifconfig en2/en3/en4 down, the interfaces showing neither UP nor
RUNNING and IP ping over the point to point link at 100% loss, the four rank
collective kept running at full speed. RDMA over Thunderbolt does not go through
the IP stack once queue pairs are established. The /30 addresses only bootstrap the
side channel.
Comparison with ring
Ring had the same shape and it was addressed: #4060 made a lost peer fail the pending
tasks, and #3742 turns that into a catchable error. On the three revisions, a two
rank ring peer loss gives: v0.32.0 hangs, current main aborts with exit 134, #3742
raises a catchable RuntimeError. JACCL has none of that.
What would help
Anything that ends the wait. A timeout on the completion queue poll, or a liveness
check on the side channel, with the group reporting the loss instead of spinning.
The recovery behaviour after SIGCONT suggests the wait itself is sound and only the
unbounded part is the problem.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with JACCL's completion-queue wait and side-channel liveness path, then compare the behavior and fixes described in ring issues #4060 and #3742. Reproduce the SIGSTOP scenario with the four-rank all_sum loop. Done means a lost peer ends the wait and reports an error instead of spinning indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100