JuliaClimate / JuliaClimate/ClimateBase.jl
ERA5 ncwrite fails
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 38
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
Tried to save a ClimArray containing ERA5 tasmax data and it failed due to a potential wrong _FillValue.
I can provide the file, but it is ~750MB. I do not remember where and when I downloaded it.
julia> fileERA5 = "/media/proy/Data/DATA/ERA5_tasmax.nc"
julia> A = ncread(fileERA5, "t2m")
julia> A
ClimArray (named t2m) with dimensions:
Longitude (type Lon): 0.0f0:0.25f0:359.75f0 (Sampled: Ordered Regular Points)
Latitude (type Lat): 90.0f0:-0.25f0:-90.0f0 (Sampled: Ordered Regular Points)
Time (type Ti): DateTime[2000-01-01T00:00:00, 2000-01-02T00:00:00, …, 2000-12-30T00:00:00, 2000-12-31T00:00:00] (Sampled: Ordered Irregular Points)
attributes: Dict{String, Any} with 6 entries:
"missing_value" => -32767
"units" => "K"
"add_offset" => 256.979
"long_name" => "2 metre temperature"
"scale_factor" => 0.00192051
"_FillValue" => -32767
and data: 1440×721×366 Array{Float64, 3}
[:, :, 1]
264.693 264.063 263.793 263.725 263.631 263.608 263.585 263.608 263.821 … 244.088 244.011 244.009 243.971 243.958 243.84 243.721 243.441
264.693 264.063 263.793 263.725 263.631 263.612 263.595 263.624 263.829 244.079 244.006 244.006 243.969 243.956 243.84 243.721 243.441
264.693 264.063 263.793 263.725 263.633 263.616 263.604 263.639 263.837 244.071 244.0 244.002 243.969 243.956 243.84 243.721 243.441
264.693 264.063 263.793 263.727 263.633 263.618 263.614 263.656 263.844 244.061 243.994 243.998 243.967 243.956 243.838 243.721 243.441
⋮ ⋮ ⋱ ⋮ ⋮
264.693 264.063 263.798 263.737 263.654 263.608 263.562 263.572 263.766 244.107 244.032 244.023 243.977 243.961 243.842 243.721 243.441
264.693 264.063 263.796 263.733 263.645 263.608 263.57 263.583 263.785 244.102 244.027 244.019 243.975 243.959 243.842 243.721 243.441
264.693 264.063 263.795 263.729 263.637 263.608 263.577 263.597 263.804 244.094 244.019 244.015 243.973 243.958 243.84 243.721 243.441
[and 365 more slices...]
julia > ncwrite("testClimateBase.nc", A)
processing variable t2m...
writing dimension lon...
writing dimension lat...
writing dimension time...
writing the CF-variable...
ERROR: NetCDF error: NetCDF: Not a valid data type or _FillValue type mismatch (NetCDF error code: -45)
Stacktrace:
[1] check
@ ~/.julia/packages/NCDatasets/eDnxY/src/errorhandling.jl:33 [inlined]
[2] nc_put_att(ncid::Int32, varid::Int32, name::String, data::Int16)
@ NCDatasets ~/.julia/packages/NCDatasets/eDnxY/src/netcdf_c.jl:521
[3] setindex!(a::NCDatasets.Attributes{NCDataset{Nothing}}, data::Int16, name::String)
@ NCDatasets ~/.julia/packages/NCDatasets/eDnxY/src/attributes.jl:57
[4] defVar(ds::NCDataset{Nothing}, name::String, vtype::DataType, dimnames::Tuple{String, String, String}; kwargs::Base.Iterators.Pairs{Symbol, Vector{Pair{String, Any}}, Tuple{Symbol}, NamedTuple{(:attrib,), Tuple{Vector{Pair{String, Any}}}}})
@ NCDatasets ~/.julia/packages/NCDatasets/eDnxY/src/cfvariable.jl:158
[5] _defVar(ds::NCDataset{Nothing}, name::String, data::Array{Float64, 3}, nctype::Type, dimnames::Tuple{String, String, String}; attrib::Dict{String, Any}, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ NCDatasets ~/.julia/packages/NCDatasets/eDnxY/src/cfvariable.jl:234
[6] #defVar#40
@ ~/.julia/packages/NCDatasets/eDnxY/src/cfvariable.jl:187 [inlined]
[7] ncwrite(file::String, Xs::Tuple{ClimArray{Float64, 3, Tuple{Lon{StepRangeLen{Float32, Float64, Float64}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ForwardIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Regular{Float32}, DimensionalData.Points}, Dict{String, String}}, Lat{StepRangeLen{Float32, Float64, Float64}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ReverseIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Regular{Float32}, DimensionalData.Points}, Dict{String, String}}, Ti{Vector{DateTime}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ForwardIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Irregular{DimensionalData.AutoBounds}, DimensionalData.Points}, Dict{String, String}}}, Tuple{}, Array{Float64, 3}, Dict{String, Any}}}; globalattr::Dict{Any, Any})
@ ClimateBase ~/.julia/packages/ClimateBase/klYxO/src/core/nc_io.jl:348
[8] #ncwrite#34
@ ~/.julia/packages/ClimateBase/klYxO/src/core/nc_io.jl:319 [inlined]
[9] ncwrite(file::String, X::ClimArray{Float64, 3, Tuple{Lon{StepRangeLen{Float32, Float64, Float64}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ForwardIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Regular{Float32}, DimensionalData.Points}, Dict{String, String}}, Lat{StepRangeLen{Float32, Float64, Float64}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ReverseIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Regular{Float32}, DimensionalData.Points}, Dict{String, String}}, Ti{Vector{DateTime}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ForwardIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Irregular{DimensionalData.AutoBounds}, DimensionalData.Points}, Dict{String, String}}}, Tuple{}, Array{Float64, 3}, Dict{String, Any}})
@ ClimateBase ~/.julia/packages/ClimateBase/klYxO/src/core/nc_io.jl:319
[10] top-level scope
@ REPL[113]:1
pkg> st
Status `~/Project.toml`
[35604d93] ClimateBase v0.12.3
[85f8d34a] NCDatasets v0.11.3
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/core/nc_io.jl around the ncwrite call shown in the stack trace, then follow the NCDatasets cfvariable.jl attribute handling that produces the _FillValue mismatch. Reproduce the failure with the provided ERA5 example or an equivalent ClimArray, and verify that ncwrite completes successfully without the NetCDF type-mismatch error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100