pydata / pydata/xarray

xr.open_dataset() reading ubyte variables as float32 from DAP server

Open
#7,782 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-CF conventions
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

Trying to open and save a netcdf file through CEDA's DAP server (http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc) whose variables scfv and scfv_unc are of type ubyte. File DDS is as follows:

Dataset {
    Float64 lat_bnds[lat = 18000][nv = 2];
    Float64 lon_bnds[lon = 36000][nv = 2];
    Float32 time[time = 1];
    Float64 lat[lat = 18000];
    Float64 lon[lon = 36000];
    Grid {
     ARRAY:
        Byte scfv[time = 1][lat = 18000][lon = 36000];
     MAPS:
        Float32 time[time = 1];
        Float64 lat[lat = 18000];
        Float64 lon[lon = 36000];
    } scfv;
    Grid {
     ARRAY:
        Byte scfv_unc[time = 1][lat = 18000][lon = 36000];
     MAPS:
        Float32 time[time = 1];
        Float64 lat[lat = 18000];
        Float64 lon[lon = 36000];
    } scfv_unc;
    Int32 spatial_ref;
} neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc;

And its DAS has attribute _Unsigned as true.

Attributes {
    lat_bnds {
    }
    lon_bnds {
    }
    time {
        String axis "T";
        String standard_name "time";
        String long_name "time";
        String calendar "standard";
        String units "hours since 1950-01-01 00:00:00";
        Int32 _ChunkSizes 1024;
    }
    lat {
        String axis "Y";
        String standard_name "latitude";
        String long_name "WGS84 latitude coordinates, center of pixel";
        String units "degrees_north";
        Float64 actual_range -89.995, 89.995;
        String bounds "lat_bnds";
        Int32 _ChunkSizes 18000;
    }
    lon {
        String axis "X";
        String standard_name "longitude";
        String long_name "WGS84 longitude coordinates, center of pixel";
        String units "degrees_east";
        Float64 actual_range -179.995, 179.995;
        String bounds "lon_bnds";
        Int32 _ChunkSizes 36000;
    }
    scfv {
        String _Unsigned "true";
        Int16 _FillValue -1;
        String standard_name "snow_area_fraction_viewable_from_above";
        String long_name "Snow Cover Fraction Viewable";
        String units "percent";
        Int16 valid_range 0, -2;
        Byte actual_range 0, 100;
        Int16 flag_values -51, -50, -46, -41, -4, -3, -2;
        String flag_meanings "Cloud Polar_Night_or_Night Water Permanent_Snow_and_Ice Classification_failed Input_Data_Error No_Satellite_Acquisition";
        Int16 missing_value -1;
        String ancillary_variables "scfv_unc";
        String grid_mapping "spatial_ref";
        Int32 _ChunkSizes 1, 1385, 2770;
    }
    scfv_unc {
        String _Unsigned "true";
        Int16 _FillValue -1;
        String standard_name "snow_area_fraction_viewable_from_above standard_error";
        String long_name "Unbiased Root Mean Square Error for Snow Cover Fraction Viewable";
        String units "percent";
        Int16 valid_range 0, -2;
        Byte actual_range 0, 100;
        Int16 flag_values -51, -50, -46, -41, -4, -3, -2;
        String flag_meanings "Cloud Polar_Night_or_Night Water Permanent_Snow_and_Ice Classification_failed Input_Data_Error No_Satellite_Acquisition";
        Int16 missing_value -1;
        String grid_mapping "spatial_ref";
        Int32 _ChunkSizes 1, 1385, 2770;
    }
    ...

Using xr.open_dataset(http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc) the mentioned variables get read as float32 instead of ubyte or at least byte

What did you expect to happen?

The returned Dataset should have scfv and scfv_unc of dtype ubyte

Minimal Complete Verifiable Example
import xarray as xr
ds = xr.open_dataset("http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc")
print(ds)
MVCE confirmation
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output
/usr/local/install/python-3.9/lib/python3.9/site-packages/xarray/conventions.py:523: SerializationWarning: variable 'scfv' has multiple fill values {-1, 255}, decoding all values to NaN.
  new_vars[k] = decode_cf_variable(
/usr/local/install/python-3.9/lib/python3.9/site-packages/xarray/conventions.py:523: SerializationWarning: variable 'scfv_unc' has multiple fill values {-1, 255}, decoding all values to NaN.
  new_vars[k] = decode_cf_variable(
<xarray.Dataset>
Dimensions:      (lat: 18000, nv: 2, lon: 36000, time: 1)
Coordinates:
  * time         (time) datetime64[ns] 2010-01-01
  * lat          (lat) float64 -90.0 -89.98 -89.97 -89.97 ... 89.98 89.98 90.0
  * lon          (lon) float64 -180.0 -180.0 -180.0 -180.0 ... 180.0 180.0 180.0
Dimensions without coordinates: nv
Data variables:
    lat_bnds     (lat, nv) float64 ...
    lon_bnds     (lon, nv) float64 ...
    spatial_ref  int32 ...
    scfv         (time, lat, lon) float32 ...
    scfv_unc     (time, lat, lon) float32 ...
Attributes: (12/43)
    title:                           ESA CCI viewable snow product level L3C ...
    institution:                     ENVEO IT GmbH
    source:                          TERRA MODIS, Collection 6.1: calibrated ...
    history:                         2021-12-06: ESA snow_cci processing line...
    references:                      http://snow-cci.enveo.at/
    tracking_id:                     2be7fb10-d660-497c-99b9-086372ec6c83
    ...                              ...
    platform:                        TERRA
    sensor:                          MODIS
    spatial_resolution:              0.01 degree
    key_variables:                   scfv
    doi:                             10.5285/ebe625b6f77945a68bda0ab7c78dd76b
    DODS_EXTRA.Unlimited_Dimension:  time
Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None
python: 3.9.15 (main, Nov 24 2022, 14:31:59)
[GCC 11.2.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.80.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.8.1

xarray: 2022.11.0
pandas: 1.5.2
numpy: 1.23.5
scipy: 1.9.3
netCDF4: 1.6.2
pydap: None
h5netcdf: None
h5py: 3.7.0
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.3.3
cfgrib: 0.9.10.3
iris: None
bottleneck: None
dask: 2022.11.1
distributed: None
matplotlib: 3.5.2
cartopy: 0.21.0
seaborn: 0.12.1
numbagg: None
fsspec: 2022.11.0
cupy: None
pint: 0.20.1
sparse: 0.13.0
flox: None
numpy_groupies: None
setuptools: 65.5.0
pip: 22.2.2
conda: 22.9.0
pytest: 7.2.0
IPython: 7.33.0
sphinx: 5.3.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the provided URL and the xr.open_dataset() entry point. Trace how the DAP Byte type and _Unsigned and fill-value attributes are decoded for scfv and scfv_unc. Done means those variables retain an unsigned-byte dtype rather than becoming float32.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.