JuliaDataCubes / JuliaDataCubes/YAXArrays.jl
`mapCube` in Distributed model is many times slower
Open
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 132
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
Serial: 8s
using YAXArrays, Statistics, Zarr
using DimensionalData
using Dates
axlist = (
Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-12-31")),
Dim{:lon}(range(1, 10, length=1000)),
Dim{:lat}(range(1, 5, length=1500)),
Dim{:Variable}(["var1", "var2"])
)
data = rand(axlist...)
ds = YAXArray(axlist, data)
c = ds[Variable = At("var1")] # see OpenNetCDF to get the file
c2 = setchunks(c, (365, 100, 150))
function mymean(output, xin)
# @show "doing a mean"
output[:] .= mean(xin)
end
indims = InDims("time")
outdims = OutDims()
@time resultcube = mapCube(mymean, c2; indims, outdims)
Parallel: 660 seconds
using Distributed
addprocs(4)
# @everywhere using Pkg
# @everywhere Pkg.activate(".")
@everywhere begin
# using NetCDF
using YAXArrays
using Statistics
using Zarr
end
@everywhere function mymean(output, xin)
# @show "doing a mean"
output[:] .= mean(xin)
end
indims = InDims("time")
outdims = OutDims()
# turn out to be slower? why?
@time resultcube2 = mapCube(mymean, c2; indims, outdims)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the serial and Distributed examples in the issue, focusing on the mapCube entry point and the c2 chunking setup. Compare the parallel execution and data movement with the serial run; done means identifying the cause of the slowdown and documenting or correcting the behavior with evidence from the benchmark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100