xr.open_dataset() reading ubyte variables as float32 from DAP server
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
Trying to open and save a netcdf file through CEDA's DAP server (http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc) whose variables scfv and scfv_unc are of type ubyte. File DDS is as follows:
Dataset {
Float64 lat_bnds[lat = 18000][nv = 2];
Float64 lon_bnds[lon = 36000][nv = 2];
Float32 time[time = 1];
Float64 lat[lat = 18000];
Float64 lon[lon = 36000];
Grid {
ARRAY:
Byte scfv[time = 1][lat = 18000][lon = 36000];
MAPS:
Float32 time[time = 1];
Float64 lat[lat = 18000];
Float64 lon[lon = 36000];
} scfv;
Grid {
ARRAY:
Byte scfv_unc[time = 1][lat = 18000][lon = 36000];
MAPS:
Float32 time[time = 1];
Float64 lat[lat = 18000];
Float64 lon[lon = 36000];
} scfv_unc;
Int32 spatial_ref;
} neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc;
And its DAS has attribute _Unsigned as true.
Attributes {
lat_bnds {
}
lon_bnds {
}
time {
String axis "T";
String standard_name "time";
String long_name "time";
String calendar "standard";
String units "hours since 1950-01-01 00:00:00";
Int32 _ChunkSizes 1024;
}
lat {
String axis "Y";
String standard_name "latitude";
String long_name "WGS84 latitude coordinates, center of pixel";
String units "degrees_north";
Float64 actual_range -89.995, 89.995;
String bounds "lat_bnds";
Int32 _ChunkSizes 18000;
}
lon {
String axis "X";
String standard_name "longitude";
String long_name "WGS84 longitude coordinates, center of pixel";
String units "degrees_east";
Float64 actual_range -179.995, 179.995;
String bounds "lon_bnds";
Int32 _ChunkSizes 36000;
}
scfv {
String _Unsigned "true";
Int16 _FillValue -1;
String standard_name "snow_area_fraction_viewable_from_above";
String long_name "Snow Cover Fraction Viewable";
String units "percent";
Int16 valid_range 0, -2;
Byte actual_range 0, 100;
Int16 flag_values -51, -50, -46, -41, -4, -3, -2;
String flag_meanings "Cloud Polar_Night_or_Night Water Permanent_Snow_and_Ice Classification_failed Input_Data_Error No_Satellite_Acquisition";
Int16 missing_value -1;
String ancillary_variables "scfv_unc";
String grid_mapping "spatial_ref";
Int32 _ChunkSizes 1, 1385, 2770;
}
scfv_unc {
String _Unsigned "true";
Int16 _FillValue -1;
String standard_name "snow_area_fraction_viewable_from_above standard_error";
String long_name "Unbiased Root Mean Square Error for Snow Cover Fraction Viewable";
String units "percent";
Int16 valid_range 0, -2;
Byte actual_range 0, 100;
Int16 flag_values -51, -50, -46, -41, -4, -3, -2;
String flag_meanings "Cloud Polar_Night_or_Night Water Permanent_Snow_and_Ice Classification_failed Input_Data_Error No_Satellite_Acquisition";
Int16 missing_value -1;
String grid_mapping "spatial_ref";
Int32 _ChunkSizes 1, 1385, 2770;
}
...
Using xr.open_dataset(http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc) the mentioned variables get read as float32 instead of ubyte or at least byte
What did you expect to happen?
The returned Dataset should have scfv and scfv_unc of dtype ubyte
Minimal Complete Verifiable Example
import xarray as xr
ds = xr.open_dataset("http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc")
print(ds)
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output
/usr/local/install/python-3.9/lib/python3.9/site-packages/xarray/conventions.py:523: SerializationWarning: variable 'scfv' has multiple fill values {-1, 255}, decoding all values to NaN.
new_vars[k] = decode_cf_variable(
/usr/local/install/python-3.9/lib/python3.9/site-packages/xarray/conventions.py:523: SerializationWarning: variable 'scfv_unc' has multiple fill values {-1, 255}, decoding all values to NaN.
new_vars[k] = decode_cf_variable(
<xarray.Dataset>
Dimensions: (lat: 18000, nv: 2, lon: 36000, time: 1)
Coordinates:
* time (time) datetime64[ns] 2010-01-01
* lat (lat) float64 -90.0 -89.98 -89.97 -89.97 ... 89.98 89.98 90.0
* lon (lon) float64 -180.0 -180.0 -180.0 -180.0 ... 180.0 180.0 180.0
Dimensions without coordinates: nv
Data variables:
lat_bnds (lat, nv) float64 ...
lon_bnds (lon, nv) float64 ...
spatial_ref int32 ...
scfv (time, lat, lon) float32 ...
scfv_unc (time, lat, lon) float32 ...
Attributes: (12/43)
title: ESA CCI viewable snow product level L3C ...
institution: ENVEO IT GmbH
source: TERRA MODIS, Collection 6.1: calibrated ...
history: 2021-12-06: ESA snow_cci processing line...
references: http://snow-cci.enveo.at/
tracking_id: 2be7fb10-d660-497c-99b9-086372ec6c83
... ...
platform: TERRA
sensor: MODIS
spatial_resolution: 0.01 degree
key_variables: scfv
doi: 10.5285/ebe625b6f77945a68bda0ab7c78dd76b
DODS_EXTRA.Unlimited_Dimension: time
Anything else we need to know?
No response
Environment
INSTALLED VERSIONS
commit: None
python: 3.9.15 (main, Nov 24 2022, 14:31:59)
[GCC 11.2.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.80.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.8.1
xarray: 2022.11.0
pandas: 1.5.2
numpy: 1.23.5
scipy: 1.9.3
netCDF4: 1.6.2
pydap: None
h5netcdf: None
h5py: 3.7.0
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.3.3
cfgrib: 0.9.10.3
iris: None
bottleneck: None
dask: 2022.11.1
distributed: None
matplotlib: 3.5.2
cartopy: 0.21.0
seaborn: 0.12.1
numbagg: None
fsspec: 2022.11.0
cupy: None
pint: 0.20.1
sparse: 0.13.0
flox: None
numpy_groupies: None
setuptools: 65.5.0
pip: 22.2.2
conda: 22.9.0
pytest: 7.2.0
IPython: 7.33.0
sphinx: 5.3.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the provided URL and the xr.open_dataset() entry point. Trace how the DAP Byte type and _Unsigned and fill-value attributes are decoded for scfv and scfv_unc. Done means those variables retain an unsigned-byte dtype rather than becoming float32.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100