JuliaDataCubes / JuliaDataCubes/YAXArrays.jl
out of memory when using Distributed
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 132
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
Is unclear to me why the following runs out of memory. The yaxarray that I'm using is not that big. Is it being copied/transfer to each worker? If so, it seems to be inefficient and using a similar approach to SharedArrays[is not like that already? 😕] probably will be better, if it is remotely an option.
Example adapted from here.
using Distributed
addprocs(7)
@everywhere using Pkg
@everywhere Pkg.activate(".")
@everywhere using YAXArrays
@everywhere using Statistics
@everywhere using Zarr
@everywhere function mymean(output, pixel)
output = mean(pixel)
end
axlist = [
RangeAxis("time", range(1, 20, length=2000)),
RangeAxis("x", range(1, 10, length=200)),
RangeAxis("y", range(1, 5, length=200)),
CategoricalAxis("Variable", ["var1", "var2"])]
data = rand(2000, 200, 200, 2);
ds = YAXArray(axlist, data)
indims = InDims("Time")
outdims = OutDims()
resultcube = mapCube(mymean, ds, indims=indims, outdims=outdims)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided Julia example with Distributed, YAXArrays, and the mapCube call to reproduce the memory growth. Investigate whether the YAXArray or its data is copied to each worker, compare that behavior with the SharedArrays approach mentioned in the issue, and document or address the confirmed cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100