Datetime accessor fails on cftime arrays with missing values
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened:
I have a computation that output dates but that sometimes also outputs missing data. (It computes the start date of a run in a timeseries, if there is no run, it outputs NaN). Afterwards, I'd like to convert those dates to dayofyear, thus I call out.dt.dayofyear. In a case where the first value of out is missing, it fails.
What you expected to happen:
I expected out.dt.dayofyear to return an array where the first value would be NaN.
Minimal Complete Verifiable Example:
import xarray as xr
import numpy as np
da = xr.DataArray(
[[np.nan, np.nan], [1, 2]],
dims=('x', 'time'),
coords={'x': [1, 2], 'time': xr.cftime_range('2000-01-01', periods=2)},
)
# out is a "object" array, where the first element is NaN
out = da.idxmin('time')
out.dt.dayofyear
# Expected : [nan, 1.]
Got:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-56-06aa9bdfd6b8> in <module>
----> 1 da.idxmin('time').dt.dayofyear
~/.conda/envs/xclim-dev/lib/python3.8/site-packages/xarray/core/utils.py in __get__(self, obj, cls)
917 return self._accessor
918
--> 919 return self._accessor(obj)
920
921
~/.conda/envs/xclim-dev/lib/python3.8/site-packages/xarray/core/accessor_dt.py in __new__(cls, obj)
514 # do all the validation here.
515 if not _contains_datetime_like_objects(obj):
--> 516 raise TypeError(
517 "'.dt' accessor only available for "
518 "DataArray with datetime64 timedelta64 dtype or "
TypeError: '.dt' accessor only available for DataArray with datetime64 timedelta64 dtype or for arrays containing cftime datetime objects.
Anything else we need to know?:
This also triggers computation when da is lazy. A lazy .dt accessor would be useful.
The laziness of it aside, would it be meaningful to change:
https://github.com/pydata/xarray/blob/d4b7a608bab0e7c140937b0b59ca45115d205145/xarray/core/common.py#L1822
to cycle on the array while np.isnan(sample) ?
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27)
[GCC 9.3.0]
python-bits: 64
OS: Linux
OS-release: 5.10.16-arch1-1
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: fr_CA.utf8
LOCALE: fr_CA.UTF-8
libhdf5: 1.10.6
libnetcdf: 4.7.4
xarray: 0.17.0
pandas: 1.0.3
numpy: 1.18.1
scipy: 1.4.1
netCDF4: 1.5.4
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.4.0
cftime: 1.3.1
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: 1.2.1
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2.12.0
distributed: 2.20.0
matplotlib: 3.3.4
cartopy: None
seaborn: None
numbagg: None
pint: 0.16.1
setuptools: 49.6.0.post20210108
pip: 21.0.1
conda: None
pytest: 5.4.3
IPython: 7.21.0
sphinx: 3.1.2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The failure is raised while validating the .dt accessor in xarray/core/accessor_dt.py, with the referenced array check in xarray/core/common.py. Reproduce the supplied cftime example, then make the accessor handle a leading missing value without losing cftime detection; verify that the result is [nan, 1.] and consider the noted lazy-array behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100