Misuse of finalizers in communicator.jl makes NCCL.jl hang/crash on any distributed job.
- Dominant language
- Julia
- Stars
- 31
- Forks
- 15
- PR merge metrics
- No merged PRs in 30d
Description
NCCL.jl incorrectly uses finalizers in [communicator.jl](https://github.com/JuliaGPU/NCCL.jl/blob/154654b65c88de7bfc8886ea97ea9bd7ee817818/src/communicator.jl#L48) on lines 48 and 73. This is the NCCL.jl version of the exact same bug in [MUMPS.jl](https://github.com/JuliaSmoothOptimizers/MUMPS.jl/issues/183). The same exact bug also exists in [CUDSS.jl](https://github.com/search?q=repo%3Aexanauts%2FCUDSS.jl%20finalizer&type=code).
This causes a crash on clusters (you need at least 2 nodes to repro this bug). See the MUMPS bug report and minimal repro to see the logic of it. Essentially, each rank will invoke finalizers in some nondeterministic order and desynchronize the cluster. I've had this happen to me systematically with just 2 ranks, 1 GPU per rank on the DMOG cluster at Heriot-Watt University.
In my opinion, the correct solution in NCCL.jl is to not use Julia finalizers to manage this and instead require the users to manually cleanup, like they do for files with open() and close().
I'm informing you as a courtesy. In my own project, I'm bypassing NCCL.jl and using NCCL_jll directly.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/communicator.jl at lines 48 and 73, then read the linked MUMPS.jl issue and minimal reproduction to understand the finalizer ordering problem. Reproduce the failure with at least two nodes or ranks, and verify that communicator cleanup no longer causes ranks to desynchronize, hang, or crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100