JuliaLang / JuliaLang/Distributed.jl
Large number of temporary shared arrays crashes Julia. Need a more aggressive gc?
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 55
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
On my computer julia always crashes on this code:
@everywhere const m = 100
@everywhere const n = 100
@everywhere const T = 100
@everywhere @inline function slice!(x, t, p)
# do something
end
function shared_test(p)
x = SharedArray(Float64, (m, n, T), init= S -> S[localindexes(S)] = 0.0)
@sync @parallel for t= 1:T
slice!(x, t, p)
end
return x
end
function doit()
p = rand(m,n,T)
for i=1: 10000000000
x = shared_test(p)
println(i)
end
end
doit()
This triggers a signal (7): Bus error which results in the workers beeing killed.
I tested this, because I wanted to isolate a memory leak that I suspect. I think it also occurs here, but Julia crashes before I can tell.
I think, if I leave out the Initialization:
x = SharedArray(Float64, (m, n, T), init= S -> S[localindexes(S)] = 0.0)
by changing the line to
x = SharedArray(Float64, (m, n, T))
there is no memory leak, but then there are other errors. I am also not sure if the parallel loop is relevant for this.
versioninfo:
Julia Version 0.4.4-pre+35
Commit 1a9429e (2016-02-12 13:21 UTC)
Platform Info:
System: Linux (x86_64-pc-linux-gnu)
CPU: AMD Phenom(tm) II X4 955 Processor
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Barcelona)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.3
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the shared_test and doit reproducer, focusing on SharedArray creation with the init callback and the @sync @parallel loop. Reproduce the Bus error using the Julia version and platform details provided, then compare behavior without initialization. Done means identifying the cause of the worker crash or memory leak and documenting a verified fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100