pydata / pydata/xarray

Datetime accessor fails on cftime arrays with missing values

Open
#5,026 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-cftime
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:
I have a computation that output dates but that sometimes also outputs missing data. (It computes the start date of a run in a timeseries, if there is no run, it outputs NaN). Afterwards, I'd like to convert those dates to dayofyear, thus I call out.dt.dayofyear. In a case where the first value of out is missing, it fails.

What you expected to happen:
I expected out.dt.dayofyear to return an array where the first value would be NaN.

Minimal Complete Verifiable Example:

import xarray as xr
import numpy as np

da = xr.DataArray(
    [[np.nan, np.nan], [1, 2]],
    dims=('x', 'time'),
    coords={'x': [1, 2], 'time': xr.cftime_range('2000-01-01', periods=2)},
)
# out is a "object" array, where the first element is NaN
out = da.idxmin('time')

out.dt.dayofyear
# Expected : [nan, 1.]

Got:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-56-06aa9bdfd6b8> in <module>
----> 1 da.idxmin('time').dt.dayofyear

~/.conda/envs/xclim-dev/lib/python3.8/site-packages/xarray/core/utils.py in __get__(self, obj, cls)
    917             return self._accessor
    918 
--> 919         return self._accessor(obj)
    920 
    921 

~/.conda/envs/xclim-dev/lib/python3.8/site-packages/xarray/core/accessor_dt.py in __new__(cls, obj)
    514         # do all the validation here.
    515         if not _contains_datetime_like_objects(obj):
--> 516             raise TypeError(
    517                 "'.dt' accessor only available for "
    518                 "DataArray with datetime64 timedelta64 dtype or "

TypeError: '.dt' accessor only available for DataArray with datetime64 timedelta64 dtype or for arrays containing cftime datetime objects.

Anything else we need to know?:
This also triggers computation when da is lazy. A lazy .dt accessor would be useful.

The laziness of it aside, would it be meaningful to change:
https://github.com/pydata/xarray/blob/d4b7a608bab0e7c140937b0b59ca45115d205145/xarray/core/common.py#L1822
to cycle on the array while np.isnan(sample) ?

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27)
[GCC 9.3.0]
python-bits: 64
OS: Linux
OS-release: 5.10.16-arch1-1
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: fr_CA.utf8
LOCALE: fr_CA.UTF-8
libhdf5: 1.10.6
libnetcdf: 4.7.4

xarray: 0.17.0
pandas: 1.0.3
numpy: 1.18.1
scipy: 1.4.1
netCDF4: 1.5.4
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.4.0
cftime: 1.3.1
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: 1.2.1
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2.12.0
distributed: 2.20.0
matplotlib: 3.3.4
cartopy: None
seaborn: None
numbagg: None
pint: 0.16.1
setuptools: 49.6.0.post20210108
pip: 21.0.1
conda: None
pytest: 5.4.3
IPython: 7.21.0
sphinx: 3.1.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The failure is raised while validating the .dt accessor in xarray/core/accessor_dt.py, with the referenced array check in xarray/core/common.py. Reproduce the supplied cftime example, then make the accessor handle a leading missing value without losing cftime detection; verify that the result is [nan, 1.] and consider the noted lazy-array behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.