JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

Netcdf _FillValue and scale_factor not applied properly when using open_dataset()

Open
#396 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

In the following netCDF:
MYD21C3_2017_cut_issue.zip

when I try:

using YAXArraysToolbox
using NetCDF
using YAXArrays
using DimensionalData
lst_monthly_high = open_dataset("/home/dpabon/Documents/MYD21C3_2017_cut_issue.nc") # years 2002 to 2011
lst_monthly_high = Cube(lst_monthly_high[["__xarray_dataarray_variable__"]])
lst_monthly_high

I got:

╭───────────────────────────────────╮
│ 7200×3600×1×12 YAXArray{UInt16,4} │
├───────────────────────────────────┴────────────────────────────────────────────────────────────── dims ┐
  ↓ lon  Sampled{Float64} -179.975:0.05000000000000001:179.97500000000005 ForwardOrdered Regular Points,
  → lat  Sampled{Float64} 89.975:-0.05:-89.97500000000002 ReverseOrdered Regular Points,
  ↗ band Sampled{Int64} 1:1 ForwardOrdered Regular Points,
  ⬔ Ti   Sampled{DateTime} [2017-01-01T00:00:00, …, 2017-12-01T00:00:00] ForwardOrdered Irregular Points
├────────────────────────────────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any} with 6 entries:
  "AREA_OR_POINT" => "Area"
  "name"          => "__xarray_dataarray_variable__"
  "add_offset"    => 0.0
  "scale_factor"  => 1.0
  "grid_mapping"  => "spatial_ref"
  "_FillValue"    => 0x0000
├───────────────────────────────────────────────────────────────────────────────────────────── file size ┤ 
  file size: 593.26 MB
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘

here "_FillValue" => 0x0000

However when I check the data:

collect(lst_monthly_high.data)


7200×3600×1×12 Array{UInt16, 4}:
[:, :, 1, 1] =
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  …  0x29e8  0x296b  0x2b25  0x28b2  0x29fe  0x29bf  0x2aa5  0x295d  0x29fd  0x28b4  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x297b  0x2975  0x29ad  0x29ef  0x2a63  0x2c07  0x0000  0x2895  0x2a9d  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x2a41  0x2a2d  0x2977  0x2891  0x2931  0x2a2d  0x0000  0x0000  0x29d2  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x2948  0x29dd  0x0000  0x2a45  0x0000  0x299d  0x2b69  0x0000  0x28b3  0x0000  0x2892
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x2a17  0x0000  0x2a1d  0x295e  0x0000  0x2a71  0x0000  0x0000  0x0000  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  …  0x28b5  0x2987  0x2951  0x299f  0x2995  0x2a84  0x28cd  0x2991  0x0000  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x29ee  0x2976  0x29c4  0x28ec  0x2a0f  0x290e  0x29f7  0x0000  0x0000  0x28ad  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x2974  0x293e  0x29eb  0x2932  0x0000  0x29fe  0x2943  0x2967  0x2844  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x29b4  0x2888  0x28c4  0x29ff  0x0000  0x2c00  0x2a0c  0x2a25  0x2a8c  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x28ed  0x2a00  0x0000  0x29c1  0x2a30  0x0000  0x0000  0x2a46  0x0000  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  …  0x28cf  0x0000  0x28c9  0x29d7  0x2abb  0x295c  0x2a5b  0x0000  0x2b6e  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x29e9  0x2a65  0x299c  0x0000  0x29cc  0x286c  0x0000  0x295c  0x29ca  0x0000  0x0000

I got 0x0000 instead of NaN or missing values. Most likely because in Julia missing and NaN cannot be Int. But also because the scale factor is not applied.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the behavior with the attached MYD21C3_2017_cut_issue.nc file and the open_dataset call shown in the issue. Inspect how open_dataset handles _FillValue and scale_factor for the loaded Julia array, then verify that missing values and scaling are represented correctly for the resulting dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.