pydata / pydata/xarray

Rolling() gives values different from pd.rolling()

Open
#5,877 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-rolling
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

I am not sure this is a bug - but it clearly doesn't give the results the user would expect.

The rolling sum of zeros gives me values that are not zeros

 var = np.array([0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.31      , 0.91999996, 8.3       ,
       1.42      , 0.03      , 1.22      , 0.09999999, 0.14      ,
       0.13      , 0.        , 0.12      , 0.03      , 2.53      ,
       0.        , 0.19999999, 0.19999999, 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ],
               dtype='float32')

timet = np.array([  43200000000000,  129600000000000,  216000000000000,  302400000000000,
        388800000000000,  475200000000000,  561600000000000,  648000000000000,
        734400000000000,  820800000000000,  907200000000000,  993600000000000,
       1080000000000000, 1166400000000000, 1252800000000000, 1339200000000000,
       1425600000000000, 1512000000000000, 1598400000000000, 1684800000000000,
       1771200000000000, 1857600000000000, 1944000000000000, 2030400000000000,
       2116800000000000, 2203200000000000, 2289600000000000, 2376000000000000,
       2462400000000000, 2548800000000000, 2635200000000000, 2721600000000000,
       2808000000000000, 2894400000000000, 2980800000000000],
      dtype='timedelta64[ns]')

ds_ex = xr.Dataset(data_vars=dict(
                          pr=(["time"], var),
                        ),
                        coords=dict(
                        time=("time", timet)
                        ),
    )

ds_ex.rolling(time=3).sum().pr.values


it gives me this result:

array([ nan, nan, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 3.1000000e-01,
1.2300000e+00, 9.5300007e+00, 1.0640000e+01, 9.7500000e+00,
2.6700001e+00, 1.3500001e+00, 1.4600002e+00, 3.7000012e-01,
2.7000013e-01, 2.5000012e-01, 1.5000013e-01, 2.6800001e+00,
2.5600002e+00, 2.7300003e+00, 4.0000033e-01, 4.0000033e-01,
2.0000035e-01, 3.5762787e-07, 3.5762787e-07, 3.5762787e-07,
3.5762787e-07, 3.5762787e-07, 3.5762787e-07, 3.5762787e-07,
3.5762787e-07, 3.5762787e-07, 3.5762787e-07
], dtype=float32)

Note the non zero values - the non zero value changes depending on whether i use float64 or float32 as precision of my data. So this seems to be a precision related issue (although the first values are correctly set to zero), in fact other sums of values are not exactly what they should be.

The small difference at the 8th/9th decimal position can be expected due to precision, but the fact that the 0s become non zeros is problematic imho, especially if not documented. Oftentimes zero in geoscience data can mean a very specific thing (i.e. zero rainfall will be characterized differently than non-zero).

in pandas this instead works:

df_ex = ds_ex.to_dataframe()
df_ex.rolling(window=3).sum().values.T

gives me

array([[ nan, nan, 0. , 0. , 0. ,
0. , 0. , 0.31 , 1.22999996, 9.53000015,
10.6400001 , 9.75000015, 2.66999999, 1.35000001, 1.46000002,
0.36999998, 0.27 , 0.24999999, 0.15 , 2.67999997,
2.55999997, 2.72999996, 0.39999998, 0.39999998, 0.19999999,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ]])

What you expected to happen:

the sum of zeros should be zero.
If this cannot be achieved/expected because of precision issues, it should be documented.

Anything else we need to know?:

I discovered this behavior in my old environments, but I created a new ad hoc environment with the latest versions, and it does the same thing.

Environment:

INSTALLED VERSIONS

commit: None
python: 3.9.7 (default, Sep 16 2021, 08:50:36)
[Clang 10.0.0 ]
python-bits: 64
OS: Darwin
OS-release: 17.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: None
libnetcdf: None

xarray: 0.19.0
pandas: 1.3.3
numpy: 1.21.2
scipy: None
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: None
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 58.0.4
pip: 21.2.4
conda: None
pytest: None
IPython: 7.28.0
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Dataset.rolling(time=3).sum() entry point and reproduce the reported float32 result using the supplied arrays. Compare it with pandas rolling behavior and determine whether the rolling sum should preserve exact zeros or whether the precision behavior needs documentation. Done means the discrepancy is resolved or clearly documented with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.