JuliaDataCubes / JuliaDataCubes/YAXArrays.jl
Netcdf _FillValue and scale_factor not applied properly when using open_dataset()
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 132
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
In the following netCDF:
MYD21C3_2017_cut_issue.zip
when I try:
using YAXArraysToolbox
using NetCDF
using YAXArrays
using DimensionalData
lst_monthly_high = open_dataset("/home/dpabon/Documents/MYD21C3_2017_cut_issue.nc") # years 2002 to 2011
lst_monthly_high = Cube(lst_monthly_high[["__xarray_dataarray_variable__"]])
lst_monthly_high
I got:
╭───────────────────────────────────╮
│ 7200×3600×1×12 YAXArray{UInt16,4} │
├───────────────────────────────────┴────────────────────────────────────────────────────────────── dims ┐
↓ lon Sampled{Float64} -179.975:0.05000000000000001:179.97500000000005 ForwardOrdered Regular Points,
→ lat Sampled{Float64} 89.975:-0.05:-89.97500000000002 ReverseOrdered Regular Points,
↗ band Sampled{Int64} 1:1 ForwardOrdered Regular Points,
⬔ Ti Sampled{DateTime} [2017-01-01T00:00:00, …, 2017-12-01T00:00:00] ForwardOrdered Irregular Points
├────────────────────────────────────────────────────────────────────────────────────────────── metadata ┤
Dict{String, Any} with 6 entries:
"AREA_OR_POINT" => "Area"
"name" => "__xarray_dataarray_variable__"
"add_offset" => 0.0
"scale_factor" => 1.0
"grid_mapping" => "spatial_ref"
"_FillValue" => 0x0000
├───────────────────────────────────────────────────────────────────────────────────────────── file size ┤
file size: 593.26 MB
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘
here "_FillValue" => 0x0000
However when I check the data:
collect(lst_monthly_high.data)
7200×3600×1×12 Array{UInt16, 4}:
[:, :, 1, 1] =
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 … 0x29e8 0x296b 0x2b25 0x28b2 0x29fe 0x29bf 0x2aa5 0x295d 0x29fd 0x28b4 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x297b 0x2975 0x29ad 0x29ef 0x2a63 0x2c07 0x0000 0x2895 0x2a9d 0x0000 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x2a41 0x2a2d 0x2977 0x2891 0x2931 0x2a2d 0x0000 0x0000 0x29d2 0x0000 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x2948 0x29dd 0x0000 0x2a45 0x0000 0x299d 0x2b69 0x0000 0x28b3 0x0000 0x2892
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x2a17 0x0000 0x2a1d 0x295e 0x0000 0x2a71 0x0000 0x0000 0x0000 0x0000 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 … 0x28b5 0x2987 0x2951 0x299f 0x2995 0x2a84 0x28cd 0x2991 0x0000 0x0000 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x29ee 0x2976 0x29c4 0x28ec 0x2a0f 0x290e 0x29f7 0x0000 0x0000 0x28ad 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x2974 0x293e 0x29eb 0x2932 0x0000 0x29fe 0x2943 0x2967 0x2844 0x0000 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x29b4 0x2888 0x28c4 0x29ff 0x0000 0x2c00 0x2a0c 0x2a25 0x2a8c 0x0000 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x28ed 0x2a00 0x0000 0x29c1 0x2a30 0x0000 0x0000 0x2a46 0x0000 0x0000 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 … 0x28cf 0x0000 0x28c9 0x29d7 0x2abb 0x295c 0x2a5b 0x0000 0x2b6e 0x0000 0x0000
0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x29e9 0x2a65 0x299c 0x0000 0x29cc 0x286c 0x0000 0x295c 0x29ca 0x0000 0x0000
I got 0x0000 instead of NaN or missing values. Most likely because in Julia missing and NaN cannot be Int. But also because the scale factor is not applied.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the behavior with the attached MYD21C3_2017_cut_issue.nc file and the open_dataset call shown in the issue. Inspect how open_dataset handles _FillValue and scale_factor for the loaded Julia array, then verify that missing values and scaling are represented correctly for the resulting dataset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100