pydata / pydata/xarray

Bottleneck and dask objects ignore `min_periods` on `rolling`

Open
#4,922 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-rolling
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:

When bottleneck is installed in an environment, it seems to ignore the min_periods kwarg on ds.rolling(...).

What you expected to happen:

When using ds.rolling(..., min_periods=1), it should be able to handle an array of length 1. Without bottleneck installed, it returns the original value of a length 1 array. With bottleneck installed, the error is:

ValueError: Moving window (=2) must between 1 and 1, inclusive

Minimal Complete Verifiable Example:

With bottleneck installed to environment:

import xarray as xr
ds = xr.DataArray([1], dims='time')
ds.rolling(time=2, center=True, min_periods=1).mean()
ValueError: Moving window (=2) must between 1 and 1, inclusive

Without bottleneck installed to environment:

import xarray as xr
ds = xr.DataArray([1], dims='time')
ds.rolling(time=2, center=True, min_periods=1).mean()

>>> <xarray.DataArray (time: 1)>
>>> array([1.])
>>> Dimensions without coordinates: time

Anything else we need to know?:

In an applied case, this came up while working on .groupby('time.dayofyear').map(_rolling), where we map a rolling mean function over a defined N days with min_periods=1. Some climatological days (like leap years) will not have the N day requirement, so the min_period catch handles that, but with bottleneck installed it breaks due to the above issue.

Environment:

Output of xr.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:22:12) [Clang 11.0.1 ] python-bits: 64 OS: Darwin OS-release: 19.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4

xarray: 0.16.2
pandas: 1.2.1
numpy: 1.19.5
scipy: 1.6.0
netCDF4: 1.5.5.1
pydap: None
h5netcdf: 0.8.1
h5py: 3.1.0
Nio: None
zarr: 2.6.1
cftime: 1.3.1
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: 1.1.8
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2021.01.1
distributed: 2021.01.1
matplotlib: 3.3.3
cartopy: 0.18.0
seaborn: 0.11.1
numbagg: None
pint: 0.16.1
setuptools: 49.6.0.post20210108
pip: 21.0
conda: None
pytest: 6.2.2
IPython: 7.18.1
sphinx: 3.4.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the minimal xarray example with and without bottleneck, then trace the rolling implementation and its bottleneck execution path. Compare both results for a length-one array with window 2 and min_periods=1; done means the bottleneck path matches the non-bottleneck result without raising the window-size error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.