JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

`mapCube` in Distributed model is many times slower

Open
#338 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

Serial: 8s

using YAXArrays, Statistics, Zarr
using DimensionalData
using Dates
axlist = (
    Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-12-31")),
    Dim{:lon}(range(1, 10, length=1000)),
    Dim{:lat}(range(1, 5, length=1500)),
    Dim{:Variable}(["var1", "var2"])
    )

data = rand(axlist...)
ds = YAXArray(axlist, data)
c = ds[Variable = At("var1")] # see OpenNetCDF to get the file
c2 = setchunks(c, (365, 100, 150))

function mymean(output, xin)
    # @show "doing a mean"
    output[:] .= mean(xin)
end

indims = InDims("time")
outdims = OutDims()
@time resultcube = mapCube(mymean, c2; indims, outdims)

image

Parallel: 660 seconds

using Distributed
addprocs(4)
# @everywhere using Pkg
# @everywhere Pkg.activate(".")
@everywhere begin
  # using NetCDF
  using YAXArrays
  using Statistics
  using Zarr
end

@everywhere function mymean(output, xin)
    # @show "doing a mean"
    output[:] .= mean(xin)
end

indims = InDims("time")
outdims = OutDims()

# turn out to be slower? why?
@time resultcube2 = mapCube(mymean, c2; indims, outdims)

image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the serial and Distributed examples in the issue, focusing on the mapCube entry point and the c2 chunking setup. Compare the parallel execution and data movement with the serial run; done means identifying the cause of the slowdown and documenting or correcting the behavior with evidence from the benchmark.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.