JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

mapCube does not work for combination of Dataset and datacube

Open
#206 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

When I want to combine a dataset with another cube in a mapCube call, I run into the dispatch of Tuples and I get the error below.
When I am using one of the data cubes in the data set instead of the whole data set, the mapCube works.
I would expect this mapCube call to do the respective computation for every subcube of the dataset, because all data cubes share the Timeaxis.

julia> freqsample
YAXArray Dataset
Dimensions: 
   Sample_5393         Axis with 3750 elements: 5393_1 5393_2 .. 5393_3749 5393_3750 
   Sample_28109        Axis with 5069 elements: 28109_1 28109_2 .. 28109_5068 28109_5069 
   Sample_27533        Axis with 5547 elements: 27533_1 27533_2 .. 27533_5546 27533_5547 
   Polarisation        Axis with 2 elements: VH VV 
   Sample_7482         Axis with 2597 elements: 7482_1 7482_2 .. 7482_2596 7482_2597 
   Time                Axis with 100 Elements from 2016-10-03T10:12:28 to 2020-02-09T10:12:44
   Sample_6187         Axis with 5595 elements: 6187_1 6187_2 .. 6187_5594 6187_5595 
   Sample_28045        Axis with 14126 elements: 28045_1 28045_2 .. 28045_14125 28045_14126 
   Sample_7555         Axis with 22929 elements: 7555_1 7555_2 .. 7555_22928 7555_22929 
   Sample_23787        Axis with 3707 elements: 23787_1 23787_2 .. 23787_3706 23787_3707 
   Sample_27588        Axis with 6924 elements: 27588_1 27588_2 .. 27588_6923 27588_6924 
   Sample_10016        Axis with 4324 elements: 10016_1 10016_2 .. 10016_4323 10016_4324 
Variables: ID_5393 ID_6187 ID_7482 ID_7555 ID_10016 ID_23787 ID_27533 ID_27588 ID_28045 ID_28109 

julia> freqsamplenorm = mapCube((x,y)->x ./ y, (freqsample, orgmean), indims=(InDims(:Time), InDims(:Time)), outdims=OutDims(:Time), inplace=false)
ERROR: MethodError: no method matching ndims(::Dataset)
Closest candidates are:
  ndims(::Union{HDF5.Attribute, HDF5.Dataset}) at ~/.julia/packages/HDF5/NqGY2/src/dataspaces.jl:257
  ndims(::DimensionalData.Dimensions.Dimension{<:AbstractArray}) at ~/.julia/packages/DimensionalData/K9D4P/src/Dimensions/dimension.jl:253
  ndims(::DimensionalData.Dimensions.Dimension) at ~/.julia/packages/DimensionalData/K9D4P/src/Dimensions/dimension.jl:252
  ...
Stacktrace:
  [1] dimnames(x::Dataset)
    @ YAXArrayBase ~/.julia/dev/YAXArrayBase/src/axisarrays/axisinterface.jl:46
  [2] caxes(x::Dataset)
    @ YAXArrays.Cubes ~/.julia/dev/YAXArrays/src/Cubes/Cubes.jl:171
  [3] findAxis(desc::YAXArrays.Cubes.Axes.ByName, c::Dataset)
    @ YAXArrays.Cubes.Axes ~/.julia/dev/YAXArrays/src/Cubes/Axes.jl:323
  [4] _broadcast_getindex_evalf
    @ ./broadcast.jl:670 [inlined]
  [5] _broadcast_getindex
    @ ./broadcast.jl:643 [inlined]
  [6] (::Base.Broadcast.var"#29#30"{Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple}, Nothing, typeof(YAXArrays.Cubes.Axes.findAxis), Tuple{Tuple{YAXArrays.Cubes.Axes.ByName}, Base.RefValue{Dataset}}}})(k::Int64)
    @ Base.Broadcast ./broadcast.jl:1075
  [7] ntuple
    @ ./ntuple.jl:48 [inlined]
  [8] copy(bc::Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple}, Nothing, typeof(YAXArrays.Cubes.Axes.findAxis), Tuple{Tuple{YAXArrays.Cubes.Axes.ByName}, Base.RefValue{Dataset}}})
    @ Base.Broadcast ./broadcast.jl:1075
  [9] materialize
    @ ./broadcast.jl:860 [inlined]
 [10] YAXArrays.DAT.InputCube(c::Dataset, desc::InDims)
    @ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:76
 [11] (::YAXArrays.DAT.var"#30#33")(o::Tuple{Dataset, InDims})
    @ YAXArrays.DAT ./none:0
 [12] iterate
    @ ./generator.jl:47 [inlined]
 [13] collect(itr::Base.Generator{Base.Iterators.Zip{Tuple{Tuple{Dataset, YAXArray{Union{Missing, Float32}, 2, Matrix{Union{Missing, Float32}}, Vector{CubeAxis}}}, Tuple{InDims, InDims}}}, YAXArrays.DAT.var"#30#33"})
    @ Base ./array.jl:787
 [14] YAXArrays.DAT.DATConfig(cdata::Tuple{Dataset, YAXArray{Union{Missing, Float32}, 2, Matrix{Union{Missing, Float32}}, Vector{CubeAxis}}}, indims::Tuple{InDims, InDims}, outdims::OutDims, inplace::Bool, max_cache::Float64, fu::Function, ispar::Bool, include_loopvars::Bool, allow_irregular::Bool, nthreads::Vector{Int64}, addargs::Tuple{}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:266
 [15] mapCube(::var"#43#44", ::Tuple{Dataset, YAXArray{Union{Missing, Float32}, 2, Matrix{Union{Missing, Float32}}, Vector{CubeAxis}}}; max_cache::Float64, indims::Tuple{InDims, InDims}, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, irregular_loopranges::Bool, nthreads::Vector{Int64}, loopchunksize::Dict{Any, Any}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:434
 [16] top-level scope
    @ REPL[232]:1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the mapCube call from the issue, then trace Dataset handling through YAXArrays/src/DAT/DAT.jl and the reported YAXArrayBase/src/axisarrays/axisinterface.jl, YAXArrays/src/Cubes/Cubes.jl, and YAXArrays/src/Cubes/Axes.jl locations. Done means mapCube accepts the Dataset alongside the other cube and performs the computation for each subcube sharing the Time axis.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.