JuliaLang / JuliaLang/Distributed.jl

SharedArray not working on remote machines

Open
#32 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
55
Forks
19
PR merge metrics
No merged PRs in 30d

Description

I am trying to set up SharedArrays on remote machines. I.e. shared among processes on the same machine. Unfortunately this doesn't seem to work. I am using version 0.5.0-dev+749 of julia.

julia> addprocs(1) #add one process on the master node
1-element Array{Int64,1}:
 2

julia> println(procs())
[1,2]

julia> S = SharedArray(Int, (3,4), init = S -> S[localindexes(S)] = myid(), pids=Int[1,2])
3x4 SharedArray{Int64,2}:
 1  1  2  2
 1  1  2  2
 1  1  2  2

julia> using ClusterManagers

julia> remotes = addprocs(SlurmManager(2), nodes=1)
srun: job 1828134 queued and waiting for resources
srun: job 1828134 has been allocated resources
2-element Array{Int64,1}:t of 2
 3
 4

julia> for w in remotes #both remote processes are on the same machine
           println(remotecall_fetch(readall, w, `hostname`)) 
       end
mrc-bsu-tesla1

mrc-bsu-tesla1


julia> r = @spawnat remotes[1] S = SharedArray(Int, (3,4), init = S -> S[localindexes(S)] = myid(), pids=remotes)
RemoteRef{Channel{Any}}(3,1,14)

julia> fetch(r)
3x4 SharedArray{Int64,2}:
 #undef  #undef  #undef  #undef
 #undef  #undef  #undef  #undef
 #undef  #undef  #undef  #undef

julia> r = @spawnat remotes[1] S*eye(4) #convert to regular array
RemoteRef{Channel{Any}}(3,1,16)

julia> fetch(r)
ERROR: On worker 3:
UndefRefError: access to undefined reference
 [inlined code] from sharedarray.jl:294
 in copy_transpose! at abstractarray.jl:513
 in copy_transpose! at linalg/matmul.jl:349
 in generic_matmatmul! at linalg/matmul.jl:459
 in * at linalg/matmul.jl:144
 in anonymous at multi.jl:1330
 in anonymous at multi.jl:889
 in run_work_thunk at multi.jl:645
 in run_work_thunk at multi.jl:654
 in anonymous at task.jl:54
 in remotecall_fetch at multi.jl:731
 [inlined code] from multi.jl:368
 in call_on_owner at multi.jl:776
 in fetch at multi.jl:784

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the remote SharedArray setup with ClusterManagers and two processes on the same machine. Start at sharedarray.jl:294 and inspect the failing access shown in the stack trace. Done means SharedArray initialization and subsequent operations work for remote processes on the same machine without undefined references.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.