JuliaLang / JuliaLang/Distributed.jl
memory is not being released aggressively enough during parallel processing
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 55
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
I have a large codebase that runs without any problems in 0.4.0, but shows symptoms of memory leak when running in 0.4.3.
Basically, the memory increases after a call to @sync @parallel for. It seems that some of the memory allocated for the parallel processing is not freed. A call to gc() in the end of the loop does not fixes the problem.
The figure below shows the memory allocation for the same script using 0.4.0 (first half of the graphic) and 0.4.3, where it can be seen thtat the memory increases after each call to @sync @parallel for.
It is the first time I fill an issue, so if I am not using the proper words/methodology, I apologize.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the memory growth around repeated @sync and @parallel calls, comparing Julia 0.4.0 with 0.4.3 as described. Trace the distributed parallel-processing path and verify completion when memory no longer increases after iterations or an explicit gc() call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100