JuliaLang / JuliaLang/Distributed.jl
printing of worker information not correct
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 55
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
using Distributed on a Slurm cluster, I am unable to connect to the workers. The reason is that SlurmClusterManager requires the output from start_worker to be in the form julia_worker:PORT#IP.IP.IP.IP. However, when using srun to launch workers across allocated resources, this is what I get:
┌ Debug: srun command: `srun -D /home/affans /home/affans/.julia/juliaup/julia-1.10.3+0.x64.linux.gnu/bin/julia --worker`
└ @ SlurmClusterManager REPL[7]:25
julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:9427#172.16.1.26
9431#172.1.1.26
9425#172.1.1.26
9428#172.1.1.26
9430#172.1.1.26
9429#172.1.1.26
9423#172.1.1.26
9432#172.1.1.26
9426#172.1.1.26
9424#172.1.1.26
So somehow the print statements (to stdout) are in a race? I asked for 10 workers here, and it seemed to print all 10 julia_workers all on the same line. Here is another example of the print :
[ Info: Worker 1 output: julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_wor$[ Info: Worker 2 output: julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_wor$[ Info: Worker 3 output: julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_wor$[ Info: Worker 4 output: 9369#172.16.1.41
[ Info: Worker 5 output: julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_worker:julia_wor$[ Info: Worker 6 output: 9361#172.16.1.41
[ Info: Worker 7 output: julia_worker:9360#172.16.1.41
[ Info: Worker 8 output: 9365#172.16.1.41
Any reason what could be causing this?
Version info:
julia> versioninfo()
Julia Version 1.10.3
Commit 0b4590a5507 (2024-04-30 10:59 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 32 × Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, broadwell)
Threads: 32 default, 0 interactive, 16 GC (on 32 virtual cores)
Environment:
LD_LIBRARY_PATH = /cm/shared/apps/slurm/16.05.8/lib64/slurm:/cm/shared/apps/slurm/16.05.8/lib64:/cm/shared/apps/openmpi/gcc/64/1.10.1/lib64
JULIA_NUM_THREADS = 32
LD_RUN_PATH = /cm/shared/apps/openmpi/gcc/64/1.10.1/lib64
julia> Distributed.VERSION
v"1.10.3"
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by tracing SlurmClusterManager's start_worker output handling and the srun worker-launch path, then reproduce the multi-worker output shown here. Done means worker announcements remain individually parseable in the expected julia_worker:PORT#IP format and workers can connect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100