JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

out of memory when using Distributed

Open
#152 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

Is unclear to me why the following runs out of memory. The yaxarray that I'm using is not that big. Is it being copied/transfer to each worker? If so, it seems to be inefficient and using a similar approach to SharedArrays[is not like that already? 😕] probably will be better, if it is remotely an option.
Example adapted from here.

using Distributed
addprocs(7)

@everywhere using Pkg
@everywhere Pkg.activate(".")
@everywhere using YAXArrays
@everywhere using Statistics
@everywhere using Zarr
@everywhere function mymean(output, pixel)
       output = mean(pixel)
end

axlist = [
    RangeAxis("time", range(1, 20, length=2000)),
    RangeAxis("x", range(1, 10, length=200)),
    RangeAxis("y", range(1, 5, length=200)),
    CategoricalAxis("Variable", ["var1", "var2"])]
data = rand(2000, 200, 200, 2);
ds = YAXArray(axlist, data)

indims = InDims("Time")
outdims = OutDims()

resultcube = mapCube(mymean, ds, indims=indims, outdims=outdims)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Julia example with Distributed, YAXArrays, and the mapCube call to reproduce the memory growth. Investigate whether the YAXArray or its data is copied to each worker, compare that behavior with the SharedArrays approach mentioned in the issue, and document or address the confirmed cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.