JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

Cube of dataset with a 0-dimensional subcube gives confusing error message

Open
#218 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

I was trying to write up an example for the JuliaClimate meta issue about julia packages for the CMIP6 data
https://github.com/JuliaClimate/meta/issues/17 and I ran into the following error with trying to convert a CMIP6 dataset into a YAXArray Cube.
The problem is, that the dataset I randomly selected seem to have a zero dimensional subcube which can't be used to derive the size of the expected data cube from the concatenation.

julia> using AWS, Zarr, YAXArrays, CSV, DataFrames

julia> AWS.global_aws_config(AWSConfig(creds=nothing, region=""))
AWSConfig(nothing, "", "json")

julia> # table with all the URLs to the data on google cloud
       CMIP6_stores = CSV.read(download("https://storage.googleapis.com/cmip6/cmip6-zarr-consolidated-stores.csv"), DataFrame)

julia> store = filter(CMIP6_stores) do row
         row.activity_id == "ScenarioMIP" && row.institution_id=="DKRZ" && row.variable_id=="tas" && row.experiment_id=="ssp585"
       end
3×11 DataFrame
 Row │ activity_id  institution_id  source_id      experiment_id  member_id  table_id  variable_id  grid_label  zstore                             dcpp_init_year  version  
     │ String15     String31        String31       String31       String15   String15  String31     String7     String                             Float64?        Int64    
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ ScenarioMIP  DKRZ            MPI-ESM1-2-HR  ssp585         r1i1p1f1   day       tas          gn          gs://cmip6/CMIP6/ScenarioMIP/DKR…         missing  20190710
   2 │ ScenarioMIP  DKRZ            MPI-ESM1-2-HR  ssp585         r1i1p1f1   Amon      tas          gn          gs://cmip6/CMIP6/ScenarioMIP/DKR…         missing  20190710
   3 │ ScenarioMIP  DKRZ            MPI-ESM1-2-HR  ssp585         r1i1p1f1   3hr       tas          gn          gs://cmip6/CMIP6/ScenarioMIP/DKR…         missing  20190710

julia> zopen(store.

activity_id     dcpp_init_year  experiment_id   grid_label      institution_id  member_id       source_id       table_id        variable_id     version         zstore
julia> zopen(store.zstore[1])
ZarrGroup at GCStore("cmip6") and path CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp585/r1i1p1f1/day/tas/gn/v20190710/
Variables: lat height time tas lat_bnds lon_bnds lon time_bnds 

julia> zopen(store.zstore[1])
ZarrGroup at GCStore("cmip6") and path CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp585/r1i1p1f1/day/tas/gn/v20190710/
Variables: lat height time tas lat_bnds lon_bnds lon time_bnds 

julia> @time zopen(store.zstore[1])
  1.190920 seconds (6.51 k allocations: 692.555 KiB)
ZarrGroup at GCStore("cmip6") and path CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp585/r1i1p1f1/day/tas/gn/v20190710/
Variables: lat height time tas lat_bnds lon_bnds lon time_bnds 

julia> c = Cube(store.zstore[1])
ERROR: ArgumentError: invalid index: nothing of type Nothing
Stacktrace:
  [1] to_index(i::Nothing)
    @ Base ./indices.jl:300
  [2] to_index(A::Vector{Any}, i::Nothing)
    @ Base ./indices.jl:277
  [3] to_indices
    @ ./indices.jl:333 [inlined]
  [4] to_indices
    @ ./indices.jl:325 [inlined]
  [5] getindex
    @ ./abstractarray.jl:1241 [inlined]
  [6] concatenatecubes(cl::Vector{Any}, cataxis::CategoricalAxis{String, :Variable, Vector{String}})
    @ YAXArrays.Cubes ~/.julia/packages/YAXArrays/Fe7F8/src/Cubes/TransformedCubes.jl:31
  [7] Cube(ds::Dataset; joinname::String)
    @ YAXArrays.Datasets ~/.julia/packages/YAXArrays/Fe7F8/src/DatasetAPI/Datasets.jl:324
  [8] Cube
    @ ~/.julia/packages/YAXArrays/Fe7F8/src/DatasetAPI/Datasets.jl:299 [inlined]
  [9] #Cube#122
    @ ~/.julia/packages/YAXArrays/Fe7F8/src/DatasetAPI/Datasets.jl:744 [inlined]
 [10] Cube(s::String)
    @ YAXArrays.Datasets ~/.julia/packages/YAXArrays/Fe7F8/src/DatasetAPI/Datasets.jl:744
 [11] top-level scope
    @ REPL[15]:1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the CMIP6 dataset example, then inspect concatenatecubes in src/Cubes/TransformedCubes.jl and Cube in src/DatasetAPI/Datasets.jl, as shown in the stack trace. Determine how the zero-dimensional subcube reaches the invalid index operation. Done means the input produces a clear error explaining the unsupported zero-dimensional subcube instead of invalid index: nothing.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.