xarray rolling does not match pandas when using min_periods and reduce
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
MCVE Code Sample
MCVE
import numpy as np
import pandas as pd
import xarray
def custom(x, axis=0):
return np.mean(x, axis)
d = pd.DataFrame(np.random.rand(100,3))
r = d.rolling(10, min_periods=5).apply(custom)
print(r.iloc[0:10,:])
xd = d.to_xarray().to_array()
r = xd.rolling(index=10, min_periods=5).reduce(custom)
print(r[:,0:10])
r = xd.rolling(index=10, min_periods=1).reduce(custom)
print(r[:,0:10])
Problem Description
I am applying a custom function on rolling windows with specific min_periods. The output of pandas..rolling.apply matches what I expect; however, the output of xarray..rolling.reduce doesn't seem to take min_periods into account.
Expected Output and Actual Output
0 1 2
0 NaN NaN NaN
1 NaN NaN NaN
2 NaN NaN NaN
3 NaN NaN NaN
4 0.632168 0.523669 0.543643
5 0.558694 0.565781 0.481204
6 0.559343 0.541787 0.415490
7 0.613457 0.554888 0.398999
8 0.579552 0.496799 0.397681
9 0.562591 0.525096 0.416461
<xarray.DataArray (variable: 3, index: 10)>
array([[ nan, nan, nan, nan, nan, nan, nan,
nan, nan, 0.562591],
[ nan, nan, nan, nan, nan, nan, nan,
nan, nan, 0.525096],
[ nan, nan, nan, nan, nan, nan, nan,
nan, nan, 0.416461]])
Coordinates:
* index (index) int64 0 1 2 3 4 5 6 7 8 9
* variable (variable) int64 0 1 2
<xarray.DataArray (variable: 3, index: 10)>
array([[ nan, nan, nan, nan, nan, nan, nan,
nan, nan, 0.562591],
[ nan, nan, nan, nan, nan, nan, nan,
nan, nan, 0.525096],
[ nan, nan, nan, nan, nan, nan, nan,
nan, nan, 0.416461]])
Coordinates:
* index (index) int64 0 1 2 3 4 5 6 7 8 9
* variable (variable) int64 0 1 2
Output of xr.show_versions()
xarray: 0.12.1
pandas: 0.24.2
numpy: 1.16.4
scipy: 1.2.1
netCDF4: 1.4.2
pydap: None
h5netcdf: None
h5py: 2.9.0
Nio: None
zarr: None
cftime: 1.0.3.4
nc_time_axis: None
PseudonetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.2.1
dask: 2.0.0
distributed: 2.0.1
matplotlib: 3.1.0
cartopy: None
seaborn: 0.9.0
setuptools: 41.0.1
pip: 19.1.1
conda: None
pytest: None
IPython: 7.5.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the MCVE using pandas rolling.apply and xarray rolling.reduce with custom and different min_periods values. Start at the rolling reduce entry point, then verify that the xarray results apply the requested minimum valid-window count and match the expected output shown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100