JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

Chunking changed for function over time axis

Open
#103 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

The following function seems to destroy the chunking of my data when it is used on a larger dataset.
It keeps the chunking, when it is applied on a small testset.

function cube_eemd(xout, xin)
    # libeemd segfaults when using Float32 data
    ind = .!ismissing.(xin)
    ts = Float64.(collect(xin[ind]))
    xout[ind] .= ceemdan(ts, 3)[:, end]
    xout[.!ind] .=missing
end

function eemdfilter(cube)
    indims = InDims("Time")
    outdims = OutDims("Time")

    efcube = mapCube(cube_eemd, cube, indims=indims, outdims=outdims, max_cache=1e8)

    newcube = mapCube(copyto!, efcube, indims=InDims("Lon", "Lat"), outdims=OutDims("Lon", "Lat"), max_cache=1e8)

    return newcube
end

This was reported as https://github.com/esa-esdl/ESDL.jl/issues/127.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the larger-dataset case in the eemdfilter function, focusing on the two mapCube calls and the cube_eemd callback. Compare chunking before and after applying the function, and use the small test set as a baseline. Done means the function preserves chunking for larger datasets as well as small ones.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.