pydata / pydata/xarray

DataArray.resample().apply() fails to apply custom function

Open
#6,953 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-faq
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

Hi,
I try to apply a custom function to aggregate a resampled object via .apply().
Maybe it is a documentation issue? But I couldn't find it.

For example calculating the median by passing np.median fails with an error as shown in the log below.

What did you expect to happen?

I would expect the median or any other custom function to be calculated for the resampled data.
It seems to work with pure pandas.

Minimal Complete Verifiable Example
import numpy as np
import pandas as pd
import xarray as xr

idx = pd.date_range("2000-01-01", "2000-12-31")
data = xr.DataArray(np.random.randn(len(idx)), coords={"index": idx})

data.resample(index="M").apply(np.median)
MVCE confirmation
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [425], in <cell line: 3>()
      1 idx = pd.date_range("2000-01-01", "2000-12-31")
      2 data = pd.Series(np.random.randn(len(idx)), index=idx).to_xarray()
----> 3 data.resample(index="M", label="left", loffset="15d").apply(np.median)

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/resample.py:279, in DataArrayResample.apply(self, func, args, shortcut, **kwargs)
    267 """
    268 Backward compatible implementation of ``map``
    269 
   (...)
    272 DataArrayResample.map
    273 """
    274 warnings.warn(
    275     "Resample.apply may be deprecated in the future. Using Resample.map is encouraged",
    276     PendingDeprecationWarning,
    277     stacklevel=2,
    278 )
--> 279 return self.map(func=func, shortcut=shortcut, args=args, **kwargs)

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/resample.py:253, in DataArrayResample.map(self, func, args, shortcut, **kwargs)
    210 """Apply a function to each array in the group and concatenate them
    211 together into a new array.
    212 
   (...)
    249     The result of splitting, applying and combining this array.
    250 """
    251 # TODO: the argument order for Resample doesn't match that for its parent,
    252 # GroupBy
--> 253 combined = super().map(func, shortcut=shortcut, args=args, **kwargs)
    255 # If the aggregation function didn't drop the original resampling
    256 # dimension, then we need to do so before we can rename the proxy
    257 # dimension we used.
    258 if self._dim in combined.coords:

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/groupby.py:1095, in DataArrayGroupByBase.map(self, func, args, shortcut, **kwargs)
   1093 grouped = self._iter_grouped_shortcut() if shortcut else self._iter_grouped()
   1094 applied = (maybe_wrap_array(arr, func(arr, *args, **kwargs)) for arr in grouped)
-> 1095 return self._combine(applied, shortcut=shortcut)

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/groupby.py:1115, in DataArrayGroupByBase._combine(self, applied, shortcut)
   1113 """Recombine the applied objects like the original."""
   1114 applied_example, applied = peek_at(applied)
-> 1115 coord, dim, positions = self._infer_concat_args(applied_example)
   1116 if shortcut:
   1117     combined = self._concat_shortcut(applied, dim, positions)

File ~/Work/miniconda3/envs/20211123_py39/lib/python3.9/site-packages/xarray/core/groupby.py:559, in GroupBy._infer_concat_args(self, applied_example)
    558 def _infer_concat_args(self, applied_example):
--> 559     if self._group_dim in applied_example.dims:
    560         coord = self._group
    561         positions = self._group_indices

AttributeError: 'numpy.float64' object has no attribute 'dims'
Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None
python: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21)
[GCC 10.3.0]
python-bits: 64
OS: Linux
OS-release: 4.4.0-210-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C
LOCALE: ('en_GB', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.8.1

xarray: 2022.6.0
pandas: 1.4.3
numpy: 1.23.2
scipy: 1.9.0
netCDF4: 1.6.0
pydap: installed
h5netcdf: None
h5py: 3.7.0
Nio: None
zarr: 2.12.0
cftime: 1.6.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2022.6.1
distributed: 2022.6.1
matplotlib: 3.5.3
cartopy: 0.20.3
seaborn: None
numbagg: None
fsspec: 2022.7.1
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 65.2.0
pip: 22.2.2
conda: 4.14.0
pytest: 7.1.2
IPython: 8.4.0
sphinx: 5.1.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the minimal example with DataArrayResample.apply() and inspect the apply/map entry points in xarray/core/resample.py, followed into the combination logic in xarray/core/groupby.py. Confirm that applying np.median to resampled data returns the expected resampled DataArray rather than the reported AttributeError, and add regression coverage for this behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.